You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "vinooganesh (via GitHub)" <gi...@apache.org> on 2026/02/17 23:15:49 UTC
[PR] [WIP] Parquet Java ALP Implementation [parquet-java]
vinooganesh opened a new pull request, #3397:
URL: https://github.com/apache/parquet-java/pull/3397
cc @julienledem @alamb
### Rationale for this change
Reworks the ALP encoding implementation to address emkornfield's architectural feedback on PR #3390. The original buffered all values in memory and decoded eagerly. This makes the writer incremental (encode per-vector as values arrive) and the reader lazy (decode on demand), matching how other Parquet encodings work.
Builds on Julien Le Dem's original implementation (https://github.com/apache/parquet-java/pull/3390). File structure, integration points, core math, and interop test infrastructure all come from his work. The rework focused on the internal writer/reader plumbing.
### What changes are included in this PR?
**Architecture (addressing review feedback):**
- Incremental writer. Values buffer in a fixed-size vector, each full vector encodes and flushes immediately.
- Lazy reader. Vectors decode on first access via offset array, skip() is O(1).
- Interleaved page layout so each vector is self-contained.
- Extracted AlpValuesReader abstract base class for shared logic.
- Preset caching. Full parameter search for first 8 vectors, top 5 combos cached for the rest.
**Spec compliance:**
- Fixed packed data size formula to ceil(n * bitWidth / 8)
- Fixed unsigned delta comparison in float writer
- Explicit little-endian byte reads instead of relying on ByteBuffer order
- Using parquet-encoding's BytePacker instead of custom bit-packing
- Capped max vector size at 32768 to prevent uint16 overflow in num_exceptions
**Code quality:**
- Renamed bitWidth overloads to prevent silent type coercion
- Package-private visibility for internals
- Configurable vector size (default 1024)
**Integration:**
- Wired ALP into DefaultV2ValuesWriterFactory and ParquetProperties
### Are these changes tested?
Yes. 105 tests across 3 test classes, all passing. Full parquet-column suite (677 tests) also passes.
Key tests construct ALP page bytes directly according to the spec and feed them to the reader without going through the writer. This verifies the reader works independently and catches any bugs where writer and reader agree with each other but disagree with the spec. Also covers NaN bit pattern preservation, negative zero roundtrip, extreme values, every partial vector remainder mod 8, skip across vector boundaries, and preset caching under distribution change.
### Are there any user-facing changes?
Users can enable ALP encoding for FLOAT and DOUBLE columns via ParquetProperties.withAlpEncoding(), globally or per-column.
Note: Likely me missing something - but @ALP is not yet in the parquet-format Thrift spec (https://github.com/apache/parquet-format/issues/533), so writing ALP files through the full Hadoop pipeline will fail at metadata serialization until parquet.thrift is updated (parquet-format PR #548).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575349278
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
+ int placeholder = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ if (!AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ placeholder = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ break;
+ }
+ }
+
+ int minValue = Integer.MAX_VALUE;
+ for (int i = 0; i < vectorLen; i++) {
+ if (AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ excPosBuffer[excIdx] = (short) i;
+ excValBuffer[excIdx] = vectorBuffer[i];
+ excIdx++;
+ encodedBuffer[i] = placeholder;
+ } else {
+ encodedBuffer[i] = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ }
+ if (encodedBuffer[i] < minValue) {
+ minValue = encodedBuffer[i];
+ }
+ }
+
+ // Subtract min so deltas start at 0, reducing bit width.
+ // The subtraction may wrap for large ranges but unsigned bits stay correct.
+ int maxDelta = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ encodedBuffer[i] = encodedBuffer[i] - minValue;
+ if (Integer.compareUnsigned(encodedBuffer[i], maxDelta) > 0) {
+ maxDelta = encodedBuffer[i];
+ }
+ }
+
+ int bitWidth = AlpEncoderDecoder.bitWidthForInt(maxDelta);
Review Comment:
Using BytesUtils.getWidthFromMaxInt here now.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
+ int placeholder = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ if (!AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ placeholder = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ break;
+ }
+ }
+
+ int minValue = Integer.MAX_VALUE;
+ for (int i = 0; i < vectorLen; i++) {
+ if (AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ excPosBuffer[excIdx] = (short) i;
+ excValBuffer[excIdx] = vectorBuffer[i];
+ excIdx++;
+ encodedBuffer[i] = placeholder;
+ } else {
+ encodedBuffer[i] = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ }
+ if (encodedBuffer[i] < minValue) {
+ minValue = encodedBuffer[i];
+ }
+ }
+
+ // Subtract min so deltas start at 0, reducing bit width.
+ // The subtraction may wrap for large ranges but unsigned bits stay correct.
+ int maxDelta = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ encodedBuffer[i] = encodedBuffer[i] - minValue;
+ if (Integer.compareUnsigned(encodedBuffer[i], maxDelta) > 0) {
+ maxDelta = encodedBuffer[i];
+ }
+ }
+
+ int bitWidth = AlpEncoderDecoder.bitWidthForInt(maxDelta);
+
+ long startSize = encodedVectors.size();
+
+ // AlpInfo: exponent(1) + factor(1) + numExceptions(2) — little-endian
+ metadataBuf[0] = (byte) params.exponent;
+ metadataBuf[1] = (byte) params.factor;
+ metadataBuf[2] = (byte) (params.numExceptions & 0xFF);
+ metadataBuf[3] = (byte) ((params.numExceptions >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, ALP_INFO_SIZE);
+
+ // ForInfo: frameOfReference(4) + bitWidth(1) — little-endian
+ metadataBuf[0] = (byte) (minValue & 0xFF);
+ metadataBuf[1] = (byte) ((minValue >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((minValue >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((minValue >>> 24) & 0xFF);
+ metadataBuf[4] = (byte) bitWidth;
+ encodedVectors.write(metadataBuf, 0, FLOAT_FOR_INFO_SIZE);
+
+ if (bitWidth > 0) {
+ packIntsWithBytePacker(encodedBuffer, vectorLen, bitWidth);
+ }
+
+ // Exception positions then values, written as separate blocks
+ if (params.numExceptions > 0) {
+ for (int i = 0; i < params.numExceptions; i++) {
+ int pos = excPosBuffer[i] & 0xFFFF;
+ metadataBuf[0] = (byte) (pos & 0xFF);
+ metadataBuf[1] = (byte) ((pos >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Short.BYTES);
+ }
+
+ for (int i = 0; i < params.numExceptions; i++) {
+ int bits = Float.floatToRawIntBits(excValBuffer[i]);
+ metadataBuf[0] = (byte) (bits & 0xFF);
+ metadataBuf[1] = (byte) ((bits >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((bits >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((bits >>> 24) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Float.BYTES);
+ }
+ }
+
+ vectorByteSizes.add((int) (encodedVectors.size() - startSize));
+ }
+
+ private void packIntsWithBytePacker(int[] values, int count, int bitWidth) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int numFullGroups = count / 8;
+ int remaining = count % 8;
+
+ for (int g = 0; g < numFullGroups; g++) {
+ packer.pack8Values(values, g * 8, packBuf, 0);
+ encodedVectors.write(packBuf, 0, bitWidth);
+ }
+
+ // Partial last group: pack 8 values (zero-padded), but only write
+ // ceil(count * bitWidth / 8) - alreadyWritten bytes per spec.
+ if (remaining > 0) {
+ System.arraycopy(values, numFullGroups * 8, packPadBuf, 0, remaining);
+ for (int i = remaining; i < 8; i++) {
+ packPadBuf[i] = 0;
+ }
+ packer.pack8Values(packPadBuf, 0, packBuf, 0);
+ int totalPackedBytes = (count * bitWidth + 7) / 8;
Review Comment:
Using BytesUtils.paddedByteCountFromBits now.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
+ int placeholder = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ if (!AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ placeholder = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ break;
+ }
+ }
+
+ int minValue = Integer.MAX_VALUE;
+ for (int i = 0; i < vectorLen; i++) {
+ if (AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ excPosBuffer[excIdx] = (short) i;
+ excValBuffer[excIdx] = vectorBuffer[i];
+ excIdx++;
+ encodedBuffer[i] = placeholder;
+ } else {
+ encodedBuffer[i] = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ }
+ if (encodedBuffer[i] < minValue) {
+ minValue = encodedBuffer[i];
+ }
+ }
+
+ // Subtract min so deltas start at 0, reducing bit width.
+ // The subtraction may wrap for large ranges but unsigned bits stay correct.
+ int maxDelta = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ encodedBuffer[i] = encodedBuffer[i] - minValue;
+ if (Integer.compareUnsigned(encodedBuffer[i], maxDelta) > 0) {
+ maxDelta = encodedBuffer[i];
+ }
+ }
+
+ int bitWidth = AlpEncoderDecoder.bitWidthForInt(maxDelta);
+
+ long startSize = encodedVectors.size();
+
+ // AlpInfo: exponent(1) + factor(1) + numExceptions(2) — little-endian
+ metadataBuf[0] = (byte) params.exponent;
+ metadataBuf[1] = (byte) params.factor;
+ metadataBuf[2] = (byte) (params.numExceptions & 0xFF);
+ metadataBuf[3] = (byte) ((params.numExceptions >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, ALP_INFO_SIZE);
+
+ // ForInfo: frameOfReference(4) + bitWidth(1) — little-endian
+ metadataBuf[0] = (byte) (minValue & 0xFF);
+ metadataBuf[1] = (byte) ((minValue >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((minValue >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((minValue >>> 24) & 0xFF);
+ metadataBuf[4] = (byte) bitWidth;
+ encodedVectors.write(metadataBuf, 0, FLOAT_FOR_INFO_SIZE);
+
+ if (bitWidth > 0) {
+ packIntsWithBytePacker(encodedBuffer, vectorLen, bitWidth);
+ }
+
+ // Exception positions then values, written as separate blocks
+ if (params.numExceptions > 0) {
+ for (int i = 0; i < params.numExceptions; i++) {
+ int pos = excPosBuffer[i] & 0xFFFF;
+ metadataBuf[0] = (byte) (pos & 0xFF);
+ metadataBuf[1] = (byte) ((pos >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Short.BYTES);
+ }
+
+ for (int i = 0; i < params.numExceptions; i++) {
+ int bits = Float.floatToRawIntBits(excValBuffer[i]);
+ metadataBuf[0] = (byte) (bits & 0xFF);
+ metadataBuf[1] = (byte) ((bits >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((bits >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((bits >>> 24) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Float.BYTES);
+ }
+ }
+
+ vectorByteSizes.add((int) (encodedVectors.size() - startSize));
+ }
+
+ private void packIntsWithBytePacker(int[] values, int count, int bitWidth) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int numFullGroups = count / 8;
+ int remaining = count % 8;
+
+ for (int g = 0; g < numFullGroups; g++) {
+ packer.pack8Values(values, g * 8, packBuf, 0);
+ encodedVectors.write(packBuf, 0, bitWidth);
+ }
+
+ // Partial last group: pack 8 values (zero-padded), but only write
+ // ceil(count * bitWidth / 8) - alreadyWritten bytes per spec.
+ if (remaining > 0) {
+ System.arraycopy(values, numFullGroups * 8, packPadBuf, 0, remaining);
+ for (int i = remaining; i < 8; i++) {
+ packPadBuf[i] = 0;
+ }
+ packer.pack8Values(packPadBuf, 0, packBuf, 0);
+ int totalPackedBytes = (count * bitWidth + 7) / 8;
+ int alreadyWritten = numFullGroups * bitWidth;
+ encodedVectors.write(packBuf, 0, totalPackedBytes - alreadyWritten);
+ }
+ }
+
+ private void buildPresetCache() {
+ // For each sampled vector, find the best (e,f) using estimated compressed size,
+ // then rank combos by how many samples they win. Take the top MAX_PRESET_COMBINATIONS.
Review Comment:
The formatting pass added spacing between the blocks here. Shout if a specific spot still reads cramped.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
+ int placeholder = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ if (!AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ placeholder = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ break;
+ }
+ }
+
+ int minValue = Integer.MAX_VALUE;
+ for (int i = 0; i < vectorLen; i++) {
+ if (AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ excPosBuffer[excIdx] = (short) i;
+ excValBuffer[excIdx] = vectorBuffer[i];
+ excIdx++;
+ encodedBuffer[i] = placeholder;
+ } else {
+ encodedBuffer[i] = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ }
+ if (encodedBuffer[i] < minValue) {
+ minValue = encodedBuffer[i];
+ }
+ }
+
+ // Subtract min so deltas start at 0, reducing bit width.
+ // The subtraction may wrap for large ranges but unsigned bits stay correct.
+ int maxDelta = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ encodedBuffer[i] = encodedBuffer[i] - minValue;
+ if (Integer.compareUnsigned(encodedBuffer[i], maxDelta) > 0) {
+ maxDelta = encodedBuffer[i];
+ }
+ }
+
+ int bitWidth = AlpEncoderDecoder.bitWidthForInt(maxDelta);
+
+ long startSize = encodedVectors.size();
+
+ // AlpInfo: exponent(1) + factor(1) + numExceptions(2) — little-endian
+ metadataBuf[0] = (byte) params.exponent;
+ metadataBuf[1] = (byte) params.factor;
+ metadataBuf[2] = (byte) (params.numExceptions & 0xFF);
+ metadataBuf[3] = (byte) ((params.numExceptions >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, ALP_INFO_SIZE);
+
+ // ForInfo: frameOfReference(4) + bitWidth(1) — little-endian
+ metadataBuf[0] = (byte) (minValue & 0xFF);
+ metadataBuf[1] = (byte) ((minValue >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((minValue >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((minValue >>> 24) & 0xFF);
+ metadataBuf[4] = (byte) bitWidth;
+ encodedVectors.write(metadataBuf, 0, FLOAT_FOR_INFO_SIZE);
+
+ if (bitWidth > 0) {
+ packIntsWithBytePacker(encodedBuffer, vectorLen, bitWidth);
+ }
+
+ // Exception positions then values, written as separate blocks
+ if (params.numExceptions > 0) {
+ for (int i = 0; i < params.numExceptions; i++) {
+ int pos = excPosBuffer[i] & 0xFFFF;
+ metadataBuf[0] = (byte) (pos & 0xFF);
+ metadataBuf[1] = (byte) ((pos >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Short.BYTES);
+ }
+
+ for (int i = 0; i < params.numExceptions; i++) {
+ int bits = Float.floatToRawIntBits(excValBuffer[i]);
+ metadataBuf[0] = (byte) (bits & 0xFF);
+ metadataBuf[1] = (byte) ((bits >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((bits >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((bits >>> 24) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Float.BYTES);
+ }
+ }
+
+ vectorByteSizes.add((int) (encodedVectors.size() - startSize));
+ }
+
+ private void packIntsWithBytePacker(int[] values, int count, int bitWidth) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int numFullGroups = count / 8;
+ int remaining = count % 8;
+
+ for (int g = 0; g < numFullGroups; g++) {
+ packer.pack8Values(values, g * 8, packBuf, 0);
+ encodedVectors.write(packBuf, 0, bitWidth);
+ }
+
+ // Partial last group: pack 8 values (zero-padded), but only write
+ // ceil(count * bitWidth / 8) - alreadyWritten bytes per spec.
+ if (remaining > 0) {
+ System.arraycopy(values, numFullGroups * 8, packPadBuf, 0, remaining);
+ for (int i = remaining; i < 8; i++) {
+ packPadBuf[i] = 0;
+ }
+ packer.pack8Values(packPadBuf, 0, packBuf, 0);
+ int totalPackedBytes = (count * bitWidth + 7) / 8;
+ int alreadyWritten = numFullGroups * bitWidth;
+ encodedVectors.write(packBuf, 0, totalPackedBytes - alreadyWritten);
+ }
+ }
+
+ private void buildPresetCache() {
Review Comment:
Storing the sampled (e,f) winners as we sample now, and buildPresetCache just does the frequency count over them, no re-search.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
+ int placeholder = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ if (!AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ placeholder = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ break;
+ }
+ }
+
+ int minValue = Integer.MAX_VALUE;
+ for (int i = 0; i < vectorLen; i++) {
+ if (AlpEncoderDecoder.isFloatException(vectorBuffer[i], params.exponent, params.factor)) {
+ excPosBuffer[excIdx] = (short) i;
+ excValBuffer[excIdx] = vectorBuffer[i];
+ excIdx++;
+ encodedBuffer[i] = placeholder;
+ } else {
+ encodedBuffer[i] = AlpEncoderDecoder.encodeFloat(vectorBuffer[i], params.exponent, params.factor);
+ }
+ if (encodedBuffer[i] < minValue) {
+ minValue = encodedBuffer[i];
+ }
+ }
+
+ // Subtract min so deltas start at 0, reducing bit width.
+ // The subtraction may wrap for large ranges but unsigned bits stay correct.
+ int maxDelta = 0;
+ for (int i = 0; i < vectorLen; i++) {
+ encodedBuffer[i] = encodedBuffer[i] - minValue;
+ if (Integer.compareUnsigned(encodedBuffer[i], maxDelta) > 0) {
+ maxDelta = encodedBuffer[i];
+ }
+ }
+
+ int bitWidth = AlpEncoderDecoder.bitWidthForInt(maxDelta);
+
+ long startSize = encodedVectors.size();
+
+ // AlpInfo: exponent(1) + factor(1) + numExceptions(2) — little-endian
+ metadataBuf[0] = (byte) params.exponent;
+ metadataBuf[1] = (byte) params.factor;
+ metadataBuf[2] = (byte) (params.numExceptions & 0xFF);
+ metadataBuf[3] = (byte) ((params.numExceptions >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, ALP_INFO_SIZE);
+
+ // ForInfo: frameOfReference(4) + bitWidth(1) — little-endian
+ metadataBuf[0] = (byte) (minValue & 0xFF);
+ metadataBuf[1] = (byte) ((minValue >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((minValue >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((minValue >>> 24) & 0xFF);
+ metadataBuf[4] = (byte) bitWidth;
+ encodedVectors.write(metadataBuf, 0, FLOAT_FOR_INFO_SIZE);
+
+ if (bitWidth > 0) {
+ packIntsWithBytePacker(encodedBuffer, vectorLen, bitWidth);
+ }
+
+ // Exception positions then values, written as separate blocks
+ if (params.numExceptions > 0) {
+ for (int i = 0; i < params.numExceptions; i++) {
+ int pos = excPosBuffer[i] & 0xFFFF;
+ metadataBuf[0] = (byte) (pos & 0xFF);
+ metadataBuf[1] = (byte) ((pos >>> 8) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Short.BYTES);
+ }
+
+ for (int i = 0; i < params.numExceptions; i++) {
+ int bits = Float.floatToRawIntBits(excValBuffer[i]);
+ metadataBuf[0] = (byte) (bits & 0xFF);
+ metadataBuf[1] = (byte) ((bits >>> 8) & 0xFF);
+ metadataBuf[2] = (byte) ((bits >>> 16) & 0xFF);
+ metadataBuf[3] = (byte) ((bits >>> 24) & 0xFF);
+ encodedVectors.write(metadataBuf, 0, Float.BYTES);
+ }
+ }
+
+ vectorByteSizes.add((int) (encodedVectors.size() - startSize));
+ }
+
+ private void packIntsWithBytePacker(int[] values, int count, int bitWidth) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int numFullGroups = count / 8;
+ int remaining = count % 8;
+
+ for (int g = 0; g < numFullGroups; g++) {
+ packer.pack8Values(values, g * 8, packBuf, 0);
+ encodedVectors.write(packBuf, 0, bitWidth);
+ }
+
+ // Partial last group: pack 8 values (zero-padded), but only write
+ // ceil(count * bitWidth / 8) - alreadyWritten bytes per spec.
+ if (remaining > 0) {
+ System.arraycopy(values, numFullGroups * 8, packPadBuf, 0, remaining);
+ for (int i = remaining; i < 8; i++) {
+ packPadBuf[i] = 0;
+ }
+ packer.pack8Values(packPadBuf, 0, packBuf, 0);
+ int totalPackedBytes = (count * bitWidth + 7) / 8;
+ int alreadyWritten = numFullGroups * bitWidth;
+ encodedVectors.write(packBuf, 0, totalPackedBytes - alreadyWritten);
+ }
+ }
+
+ private void buildPresetCache() {
+ // For each sampled vector, find the best (e,f) using estimated compressed size,
+ // then rank combos by how many samples they win. Take the top MAX_PRESET_COMBINATIONS.
+ Map<Long, Integer> comboCounts = new HashMap<>();
+ for (float[] sample : rowgroupSamples) {
+ AlpEncoderDecoder.EncodingParams best =
+ AlpEncoderDecoder.findBestFloatParams(sample, 0, sample.length);
+ long key = ((long) best.exponent << Integer.SIZE) | best.factor;
+ comboCounts.merge(key, 1, Integer::sum);
+ }
+ List<Map.Entry<Long, Integer>> sorted = new ArrayList<>(comboCounts.entrySet());
+ sorted.sort(Comparator.<Map.Entry<Long, Integer>, Integer>comparing(Map.Entry::getValue)
+ .reversed());
+ int numPresets = Math.min(sorted.size(), MAX_PRESET_COMBINATIONS);
+ cachedPresets = new int[numPresets][2];
+ for (int i = 0; i < numPresets; i++) {
+ long key = sorted.get(i).getKey();
+ cachedPresets[i][0] = (int) (key >>> Integer.SIZE); // exponent
+ cachedPresets[i][1] = (int) key; // factor
+ }
+ rowgroupSamples.clear();
+ }
+
+ @Override
+ public long getBufferedSize() {
+ // Encoded vectors so far + estimate for buffered values.
+ // 3 bytes/value ≈ ALP_INFO_SIZE(4) + FLOAT_FOR_INFO_SIZE(5) amortized over vectorSize(1024),
+ // plus ~1 bit/value for packed data at typical bit widths — rounds up to ~3 bytes.
+ return encodedVectors.size() + (long) bufferCount * 3;
+ }
+
+ @Override
+ public BytesInput getBytes() {
+ if (bufferCount > 0) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+
+ int numVectors = vectorByteSizes.size();
+
+ // Assemble: [header 7B] [offset array 4B*N] [vector data...]
+ ByteBuffer header = ByteBuffer.allocate(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ header.put((byte) ALP_COMPRESSION_MODE);
+ header.put((byte) ALP_INTEGER_ENCODING_FOR);
+ header.put((byte) logVectorSize);
+ header.putInt(totalCount);
+
+ if (totalCount == 0) {
+ return BytesInput.from(header.array());
+ }
+
+ int offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsets = ByteBuffer.allocate(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ int currentOffset = offsetArraySize;
+ for (int v = 0; v < numVectors; v++) {
+ offsets.putInt(currentOffset);
+ currentOffset += vectorByteSizes.get(v);
+ }
+
+ return BytesInput.concat(
+ BytesInput.from(header.array()), BytesInput.from(offsets.array()), BytesInput.from(encodedVectors));
+ }
+
+ @Override
+ public void reset() {
+ bufferCount = 0;
+ totalCount = 0;
+ encodedVectors.reset();
+ vectorByteSizes.clear();
+ vectorsProcessed = 0;
+ // Preserve cachedPresets across page boundaries: the preset (exponent, factor) pairs
+ // reflect the column's data distribution and remain valid for subsequent pages.
+ // Clearing them on every reset forces a full parameter search from scratch on each page,
+ // which is the main source of write overhead. Only reset the sampling state.
+ rowgroupSamples.clear();
+ }
+
+ @Override
+ public void close() {
+ encodedVectors.close();
+ }
+
+ @Override
+ public long getAllocatedSize() {
+ return (long) vectorBuffer.length * Float.BYTES + encodedVectors.getCapacity();
+ }
+
+ @Override
+ public String memUsageString(String prefix) {
+ return String.format(
+ "%s FloatAlpValuesWriter %d values, %d bytes allocated", prefix, totalCount, getAllocatedSize());
+ }
+ }
+
+ /** Double writer. Same structure as FloatAlpValuesWriter but uses longs. */
Review Comment:
Applied all of these to the double writer path too.
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -132,6 +135,8 @@ public static WriterVersion fromString(String name) {
private final int pageRowCountLimit;
private final boolean pageWriteChecksumEnabled;
private final ColumnProperty<ByteStreamSplitMode> byteStreamSplitEnabled;
+ private final ColumnProperty<Boolean> alpEnabled;
Review Comment:
Bundled into an AlpConfig, so ParquetProperties now carries one ColumnProperty<AlpConfig> instead of the two separate properties. The public setters and getters are unchanged.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpEncoderDecoderTest.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import org.junit.Test;
+
+/**
+ * Tests for the core ALP encoder/decoder logic.
+ */
+public class AlpEncoderDecoderTest {
+
+ // ========== Float Encoding/Decoding Tests ==========
+
+ @Test
+ public void testFloatRoundTrip() {
+ float[] testValues = {0.0f, 1.0f, -1.0f, 3.14159f, 100.5f, 0.001f, 1234567.0f};
+
+ for (float value : testValues) {
+ for (int exponent = 0; exponent <= AlpConstants.FLOAT_MAX_EXPONENT; exponent++) {
+ for (int factor = 0; factor <= exponent; factor++) {
+ if (!AlpEncoderDecoder.isFloatException(value, exponent, factor)) {
+ int encoded = AlpEncoderDecoder.encodeFloat(value, exponent, factor);
+ float decoded = AlpEncoderDecoder.decodeFloat(encoded, exponent, factor);
+ assertEquals(
+ "Round-trip failed for value=" + value + ", exponent=" + exponent + ", factor="
+ + factor,
+ Float.floatToRawIntBits(value),
+ Float.floatToRawIntBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testFloatExceptionDetection() {
+ assertTrue("NaN should be an exception", AlpEncoderDecoder.isFloatException(Float.NaN));
+ assertTrue(
+ "Positive infinity should be an exception",
+ AlpEncoderDecoder.isFloatException(Float.POSITIVE_INFINITY));
+ assertTrue(
+ "Negative infinity should be an exception",
+ AlpEncoderDecoder.isFloatException(Float.NEGATIVE_INFINITY));
+ assertTrue("Negative zero should be an exception", AlpEncoderDecoder.isFloatException(-0.0f));
+
+ assertFalse("1.0f should not be a basic exception", AlpEncoderDecoder.isFloatException(1.0f));
+ assertFalse("0.0f should not be a basic exception", AlpEncoderDecoder.isFloatException(0.0f));
+ }
+
+ @Test
+ public void testFloatEncoding() {
+ assertEquals(123, AlpEncoderDecoder.encodeFloat(1.23f, 2, 0));
+ assertEquals(123, AlpEncoderDecoder.encodeFloat(12.3f, 2, 1));
+ assertEquals(0, AlpEncoderDecoder.encodeFloat(0.0f, 5, 0));
+ }
+
+ @Test
+ public void testFloatDecoding() {
+ assertEquals(1.23f, AlpEncoderDecoder.decodeFloat(123, 2, 0), 1e-6f);
+ assertEquals(12.3f, AlpEncoderDecoder.decodeFloat(123, 2, 1), 1e-6f);
+ assertEquals(0.0f, AlpEncoderDecoder.decodeFloat(0, 5, 0), 0.0f);
+ }
+
+ @Test
+ public void testFloatEncodeRounding() {
+ // Verify rounding behavior (magic number trick rounds to nearest)
+ assertEquals(5, AlpEncoderDecoder.encodeFloat(5.4f, 0, 0));
+ assertEquals(6, AlpEncoderDecoder.encodeFloat(5.6f, 0, 0));
+ assertEquals(-5, AlpEncoderDecoder.encodeFloat(-5.4f, 0, 0));
+ assertEquals(-6, AlpEncoderDecoder.encodeFloat(-5.6f, 0, 0));
+ assertEquals(0, AlpEncoderDecoder.encodeFloat(0.0f, 0, 0));
+ }
+
+ @Test
+ public void testFloatEncodeDecodeWithFactor() {
+ // Verify that encode/decode with non-zero factor works correctly.
+ // The key correctness property: encode uses value * POW10[e] * POW10_NEGATIVE[f],
+ // and decode uses encoded * POW10[f] * POW10_NEGATIVE[e].
+ float value = 12.3f;
+ int encoded = AlpEncoderDecoder.encodeFloat(value, 2, 1);
+ assertEquals(123, encoded); // 12.3 * 100 * 0.1 = 123
+ float decoded = AlpEncoderDecoder.decodeFloat(encoded, 2, 1);
+ assertEquals(Float.floatToRawIntBits(value), Float.floatToRawIntBits(decoded));
+ }
+
+ // ========== Double Encoding/Decoding Tests ==========
+
+ @Test
+ public void testDoubleRoundTrip() {
+ double[] testValues = {0.0, 1.0, -1.0, 3.14159265358979, 100.5, 0.001, 12345678901234.0};
+
+ for (double value : testValues) {
+ for (int exponent = 0; exponent <= Math.min(AlpConstants.DOUBLE_MAX_EXPONENT, 10); exponent++) {
+ for (int factor = 0; factor <= exponent; factor++) {
+ if (!AlpEncoderDecoder.isDoubleException(value, exponent, factor)) {
+ long encoded = AlpEncoderDecoder.encodeDouble(value, exponent, factor);
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, exponent, factor);
+ assertEquals(
+ "Round-trip failed for value=" + value + ", exponent=" + exponent + ", factor="
+ + factor,
+ Double.doubleToRawLongBits(value),
+ Double.doubleToRawLongBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testDoubleExceptionDetection() {
+ assertTrue("NaN should be an exception", AlpEncoderDecoder.isDoubleException(Double.NaN));
+ assertTrue(
+ "Positive infinity should be an exception",
+ AlpEncoderDecoder.isDoubleException(Double.POSITIVE_INFINITY));
+ assertTrue(
+ "Negative infinity should be an exception",
+ AlpEncoderDecoder.isDoubleException(Double.NEGATIVE_INFINITY));
+ assertTrue("Negative zero should be an exception", AlpEncoderDecoder.isDoubleException(-0.0));
+
+ assertFalse("1.0 should not be a basic exception", AlpEncoderDecoder.isDoubleException(1.0));
+ assertFalse("0.0 should not be a basic exception", AlpEncoderDecoder.isDoubleException(0.0));
+ }
+
+ @Test
+ public void testDoubleEncoding() {
+ assertEquals(123L, AlpEncoderDecoder.encodeDouble(1.23, 2, 0));
+ assertEquals(123L, AlpEncoderDecoder.encodeDouble(12.3, 2, 1));
+ assertEquals(0L, AlpEncoderDecoder.encodeDouble(0.0, 5, 0));
+ }
+
+ @Test
+ public void testDoubleDecoding() {
+ assertEquals(1.23, AlpEncoderDecoder.decodeDouble(123, 2, 0), 1e-10);
+ assertEquals(12.3, AlpEncoderDecoder.decodeDouble(123, 2, 1), 1e-10);
+ assertEquals(0.0, AlpEncoderDecoder.decodeDouble(0, 5, 0), 0.0);
+ }
+
+ @Test
+ public void testDoubleEncodeRounding() {
+ // Verify rounding behavior (magic number trick rounds to nearest)
+ assertEquals(5L, AlpEncoderDecoder.encodeDouble(5.4, 0, 0));
+ assertEquals(6L, AlpEncoderDecoder.encodeDouble(5.6, 0, 0));
+ assertEquals(-5L, AlpEncoderDecoder.encodeDouble(-5.4, 0, 0));
+ assertEquals(-6L, AlpEncoderDecoder.encodeDouble(-5.6, 0, 0));
+ assertEquals(0L, AlpEncoderDecoder.encodeDouble(0.0, 0, 0));
+ }
+
+ @Test
+ public void testDoubleEncodeDecodeWithFactor() {
+ // Verify that encode/decode with non-zero factor works correctly.
+ // The key correctness property: encode uses value * POW10[e] * POW10_NEGATIVE[f],
+ // and decode uses encoded * POW10[f] * POW10_NEGATIVE[e].
+ double value = 12.3;
+ long encoded = AlpEncoderDecoder.encodeDouble(value, 2, 1);
+ assertEquals(123L, encoded); // 12.3 * 100 * 0.1 = 123
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, 2, 1);
+ assertEquals(Double.doubleToRawLongBits(value), Double.doubleToRawLongBits(decoded));
+ }
+
+ @Test
+ public void testDoubleEncodeDecodeArithmeticOrder() {
+ // This test verifies that the exact order of operations in encode/decode
+ // is critical for IEEE 754 correctness. The encode uses
+ // fastRound(value * POW10[e] * POW10_NEGATIVE[f]) and decode uses
+ // (encoded * POW10[f] * POW10_NEGATIVE[e]), both as single expressions.
+ // Splitting the multiplies or reordering changes rounding.
+ double[] testValues = {0.123456789, 1.23456789, 12.3456789, 123.456789, 1234.56789};
+ for (double value : testValues) {
+ for (int e = 0; e <= 10; e++) {
+ for (int f = 0; f <= e; f++) {
+ if (!AlpEncoderDecoder.isDoubleException(value, e, f)) {
+ long encoded = AlpEncoderDecoder.encodeDouble(value, e, f);
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, e, f);
+ assertEquals(
+ "Roundtrip failed for " + value + " (e=" + e + ", f=" + f + ")",
+ Double.doubleToRawLongBits(value),
+ Double.doubleToRawLongBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ // ========== Bit Width Tests (renamed methods) ==========
+
+ @Test
+ public void testBitWidthForInt() {
+ assertEquals(0, AlpEncoderDecoder.bitWidthForInt(0));
+ assertEquals(1, AlpEncoderDecoder.bitWidthForInt(1));
+ assertEquals(2, AlpEncoderDecoder.bitWidthForInt(2));
+ assertEquals(2, AlpEncoderDecoder.bitWidthForInt(3));
+ assertEquals(3, AlpEncoderDecoder.bitWidthForInt(4));
+ assertEquals(8, AlpEncoderDecoder.bitWidthForInt(255));
+ assertEquals(9, AlpEncoderDecoder.bitWidthForInt(256));
+ assertEquals(16, AlpEncoderDecoder.bitWidthForInt(65535));
+ assertEquals(31, AlpEncoderDecoder.bitWidthForInt(Integer.MAX_VALUE));
+ }
+
+ @Test
+ public void testBitWidthForLong() {
+ assertEquals(0, AlpEncoderDecoder.bitWidthForLong(0L));
+ assertEquals(1, AlpEncoderDecoder.bitWidthForLong(1L));
+ assertEquals(2, AlpEncoderDecoder.bitWidthForLong(2L));
+ assertEquals(2, AlpEncoderDecoder.bitWidthForLong(3L));
+ assertEquals(3, AlpEncoderDecoder.bitWidthForLong(4L));
+ assertEquals(8, AlpEncoderDecoder.bitWidthForLong(255L));
+ assertEquals(9, AlpEncoderDecoder.bitWidthForLong(256L));
+ assertEquals(16, AlpEncoderDecoder.bitWidthForLong(65535L));
+ assertEquals(31, AlpEncoderDecoder.bitWidthForLong((long) Integer.MAX_VALUE));
+ assertEquals(63, AlpEncoderDecoder.bitWidthForLong(Long.MAX_VALUE));
+ }
+
+ // ========== Best Parameters Tests ==========
+
+ @Test
+ public void testFindBestFloatParams() {
+ float[] values = {1.23f, 4.56f, 7.89f, 10.11f, 12.13f};
+ AlpEncoderDecoder.EncodingParams params = AlpEncoderDecoder.findBestFloatParams(values, 0, values.length);
+
+ assertNotNull(params);
+ assertTrue(params.exponent >= 0 && params.exponent <= AlpConstants.FLOAT_MAX_EXPONENT);
+ assertTrue(params.factor >= 0 && params.factor <= params.exponent);
+
+ for (float v : values) {
+ if (!AlpEncoderDecoder.isFloatException(v, params.exponent, params.factor)) {
+ int encoded = AlpEncoderDecoder.encodeFloat(v, params.exponent, params.factor);
+ float decoded = AlpEncoderDecoder.decodeFloat(encoded, params.exponent, params.factor);
+ assertEquals(Float.floatToRawIntBits(v), Float.floatToRawIntBits(decoded));
+ }
+ }
+ }
+
+ @Test
+ public void testFindBestFloatParamsWithAllExceptions() {
+ float[] values = {Float.NaN, Float.NaN, Float.NaN};
+ AlpEncoderDecoder.EncodingParams params = AlpEncoderDecoder.findBestFloatParams(values, 0, values.length);
+
+ assertNotNull(params);
+ assertEquals(values.length, params.numExceptions);
+ }
+
+ @Test
+ public void testFindBestDoubleParams() {
+ double[] values = {1.23, 4.56, 7.89, 10.11, 12.13};
+ AlpEncoderDecoder.EncodingParams params = AlpEncoderDecoder.findBestDoubleParams(values, 0, values.length);
+
+ assertNotNull(params);
+ assertTrue(params.exponent >= 0 && params.exponent <= AlpConstants.DOUBLE_MAX_EXPONENT);
+ assertTrue(params.factor >= 0 && params.factor <= params.exponent);
+
+ for (double v : values) {
+ if (!AlpEncoderDecoder.isDoubleException(v, params.exponent, params.factor)) {
+ long encoded = AlpEncoderDecoder.encodeDouble(v, params.exponent, params.factor);
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, params.exponent, params.factor);
+ assertEquals(Double.doubleToRawLongBits(v), Double.doubleToRawLongBits(decoded));
+ }
+ }
+ }
+
+ @Test
+ public void testFindBestDoubleParamsWithAllExceptions() {
+ double[] values = {Double.NaN, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY};
+ AlpEncoderDecoder.EncodingParams params = AlpEncoderDecoder.findBestDoubleParams(values, 0, values.length);
+
+ assertNotNull(params);
+ assertEquals(values.length, params.numExceptions);
+ }
+
+ @Test
+ public void testFindBestParamsWithOffset() {
+ float[] values = {Float.NaN, Float.NaN, 1.23f, 4.56f, 7.89f, Float.NaN};
+ AlpEncoderDecoder.EncodingParams params = AlpEncoderDecoder.findBestFloatParams(values, 2, 3);
+
+ assertNotNull(params);
+ assertEquals(0, params.numExceptions);
+ }
+
+ // ========== Preset-Based Parameter Search Tests ==========
+
+ @Test
+ public void testFindBestFloatParamsWithPresets() {
+ float[] values = {1.23f, 4.56f, 7.89f, 10.11f, 12.13f};
+ int[][] presets = {{2, 0}, {3, 0}, {4, 1}};
+ AlpEncoderDecoder.EncodingParams params =
+ AlpEncoderDecoder.findBestFloatParamsWithPresets(values, 0, values.length, presets);
+
+ assertNotNull(params);
+ // Should select one of the preset combinations
+ boolean foundMatch = false;
+ for (int[] preset : presets) {
+ if (params.exponent == preset[0] && params.factor == preset[1]) {
+ foundMatch = true;
+ break;
+ }
+ }
+ assertTrue("Result should be one of the preset combinations", foundMatch);
+ }
+
+ @Test
+ public void testFindBestDoubleParamsWithPresets() {
+ double[] values = {1.23, 4.56, 7.89, 10.11, 12.13};
+ int[][] presets = {{2, 0}, {3, 0}, {4, 1}};
+ AlpEncoderDecoder.EncodingParams params =
+ AlpEncoderDecoder.findBestDoubleParamsWithPresets(values, 0, values.length, presets);
+
+ assertNotNull(params);
+ boolean foundMatch = false;
+ for (int[] preset : presets) {
+ if (params.exponent == preset[0] && params.factor == preset[1]) {
+ foundMatch = true;
+ break;
+ }
+ }
+ assertTrue("Result should be one of the preset combinations", foundMatch);
+ }
+
+ @Test
+ public void testPresetsProduceSameResultAsFullSearch() {
+ float[] values = {1.23f, 4.56f, 7.89f};
+ AlpEncoderDecoder.EncodingParams fullResult = AlpEncoderDecoder.findBestFloatParams(values, 0, values.length);
+
+ // Include the best params in presets
+ int[][] presets = {{fullResult.exponent, fullResult.factor}, {0, 0}, {1, 0}};
+ AlpEncoderDecoder.EncodingParams presetResult =
+ AlpEncoderDecoder.findBestFloatParamsWithPresets(values, 0, values.length, presets);
+
+ assertTrue(
+ "Preset result should be at least as good as full search",
+ presetResult.numExceptions <= fullResult.numExceptions);
+ }
Review Comment:
Added findBestFloatParamsMixedSignUsesSignedDeltaRange (and a preset variant) with exactly this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3567814281
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
Review Comment:
Reworded and pushed. The comment now talks about matching the reference layout for interop instead of C++ specifically. Once ALP lands in the format spec we can point at that directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3567844556
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForFloat() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new float[capacity];
+ this.deltasBuffer = new int[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Integer.SIZE]; // max bit width for int = 32 bytes
+ }
+
+ @Override
+ public float readFloat() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP float data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > FLOAT_MAX_EXPONENT) {
+ throw new ParquetDecodingException(
+ "Invalid ALP float exponent " + exponent + " in vector " + vectorIdx + ", max is " + FLOAT_MAX_EXPONENT);
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException(
+ "Invalid ALP float factor " + factor + " > exponent " + exponent + " in vector " + vectorIdx);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException(
+ "Invalid ALP numExceptions " + numExceptions + " > vectorLen " + vectorLen + " in vector " + vectorIdx);
+ }
+
+ int frameOfReference = getIntLE(vectorsData, pos);
+ int bitWidth = vectorsData.get(pos + 4) & 0xFF;
+ pos += FLOAT_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
Review Comment:
Added a bounds check. decodeBody now throws if bitWidth is greater than 32 for the float reader.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int currentIndex;
+ protected int currentVectorIndex;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ AlpValuesReader() {
+ this.currentIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorIndex = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.currentIndex = 0;
+ this.currentVectorIndex = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ allocateDecodedBuffer(vectorSize);
+ }
+
+ protected int getVectorLength(int vectorIdx) {
+ if (vectorIdx < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorIdx) {
+ return vectorOffsets[vectorIdx] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ if (n < 0 || currentIndex + n > totalCount) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ currentIndex, n, totalCount));
+ }
+ currentIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorIdx = currentIndex / vectorSize;
+ if (vectorIdx != currentVectorIndex) {
+ decodeVector(vectorIdx);
+ currentVectorIndex = vectorIdx;
+ }
+ }
+
+ protected abstract void allocateDecodedBuffer(int capacity);
+
+ protected abstract void decodeVector(int vectorIdx);
+
+ // Explicit little-endian reads using absolute get(), since absolute get() ignores ByteBuffer order.
Review Comment:
Switched to the little-endian typed getters (getInt/getLong on a LITTLE_ENDIAN-ordered buffer) instead of the manual byte assembly, so the local LE helpers are gone.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int currentIndex;
+ protected int currentVectorIndex;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ AlpValuesReader() {
+ this.currentIndex = 0;
Review Comment:
Renamed these to make the coordinate systems clearer: pageValueIndex (position in the page), vectorNumber (which vector), and vectorSlot (position within the vector).
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
Review Comment:
Renamed to vectorNumber here too, matching the base reader.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
Review Comment:
Right, it's moot now that we read through the typed getters, so the 0xFFFF masking is gone.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > DOUBLE_MAX_EXPONENT) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double exponent " + exponent + " in vector " + vectorIdx + ", max is " + DOUBLE_MAX_EXPONENT);
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double factor " + factor + " > exponent " + exponent + " in vector " + vectorIdx);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException(
+ "Invalid ALP numExceptions " + numExceptions + " > vectorLen " + vectorLen + " in vector " + vectorIdx);
+ }
+
+ long frameOfReference = getLongLE(vectorsData, pos);
+ int bitWidth = vectorsData.get(pos + 8) & 0xFF;
Review Comment:
Using Long.BYTES now.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > DOUBLE_MAX_EXPONENT) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double exponent " + exponent + " in vector " + vectorIdx + ", max is " + DOUBLE_MAX_EXPONENT);
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double factor " + factor + " > exponent " + exponent + " in vector " + vectorIdx);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException(
+ "Invalid ALP numExceptions " + numExceptions + " > vectorLen " + vectorLen + " in vector " + vectorIdx);
+ }
+
+ long frameOfReference = getLongLE(vectorsData, pos);
+ int bitWidth = vectorsData.get(pos + 8) & 0xFF;
+ pos += DOUBLE_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
+ pos = unpackLongsWithBytePacker(vectorsData, pos, deltasBuffer, vectorLen, bitWidth);
+ } else {
+ Arrays.fill(deltasBuffer, 0, vectorLen, 0L);
+ }
+
+ for (int i = 0; i < vectorLen; i++) {
+ long encoded = deltasBuffer[i] + frameOfReference;
+ decodedValues[i] = AlpEncoderDecoder.decodeDouble(encoded, exponent, factor);
+ }
+
+ if (numExceptions > 0) {
+ for (int e = 0; e < numExceptions; e++) {
+ excPositionsBuffer[e] = getShortLE(vectorsData, pos) & 0xFFFF;
+ if (excPositionsBuffer[e] >= vectorLen) {
+ throw new ParquetDecodingException(
+ "ALP exception position " + excPositionsBuffer[e] + " out of bounds for vectorLen " + vectorLen);
+ }
+ pos += Short.BYTES;
+ }
+ for (int e = 0; e < numExceptions; e++) {
+ decodedValues[excPositionsBuffer[e]] = getDoubleLE(vectorsData, pos);
+ pos += Double.BYTES;
+ }
+ }
+ }
+
+ private int unpackLongsWithBytePacker(ByteBuffer buf, int pos, long[] output, int count, int bitWidth) {
+ BytePackerForLong packer = Packer.LITTLE_ENDIAN.newBytePackerForLong(bitWidth);
+ int numFullGroups = count / 8;
+ int remaining = count % 8;
+
+ for (int g = 0; g < numFullGroups; g++) {
+ packer.unpack8Values(buf, pos, output, g * 8);
+ pos += bitWidth;
+ }
+
+ // Last group might have fewer than 8 values; zero-pad and unpack,
+ // but only advance pos by the actual bytes in the page.
+ if (remaining > 0) {
+ int totalPackedBytes = (count * bitWidth + 7) / 8;
Review Comment:
Switched to BytesUtils.paddedByteCountFromBits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504183107
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForFloat() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new float[capacity];
+ this.deltasBuffer = new int[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Integer.SIZE]; // max bit width for int = 32 bytes
+ }
+
+ @Override
+ public float readFloat() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP float data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
Review Comment:
does it pay to pull more of the decoding into the common base class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098863710
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+final class AlpConstants {
+
+ private AlpConstants() {
+ // Utility class
+ }
+
+ // Page header fields
+ static final int ALP_COMPRESSION_MODE = 0;
+ static final int ALP_INTEGER_ENCODING_FOR = 0;
+ static final int ALP_HEADER_SIZE = 7;
+
+ static final int DEFAULT_VECTOR_SIZE = 1024;
+ static final int DEFAULT_VECTOR_SIZE_LOG = 10;
+
+ // BytePacker packs/unpacks 8 values at a time (pack8Values/unpack8Values).
+ static final int PACK_GROUP_SIZE = 8;
+
+ // Capped at 15 (vectorSize=32768) because num_exceptions is uint16,
+ // so vectorSize must not exceed 65535 to avoid overflow when all values are exceptions.
+ static final int MAX_LOG_VECTOR_SIZE = 15;
+ static final int MIN_LOG_VECTOR_SIZE = 3;
+
+ static final int FLOAT_MAX_EXPONENT = 10;
+ static final int DOUBLE_MAX_EXPONENT = 18;
+
+ // Per-vector metadata sizes in bytes
+ static final int ALP_INFO_SIZE = 4; // exponent(1) + factor(1) + num_exceptions(2)
+ static final int FLOAT_FOR_INFO_SIZE = 5; // frame_of_reference(4) + bit_width(1)
+ static final int DOUBLE_FOR_INFO_SIZE = 9; // frame_of_reference(8) + bit_width(1)
+
+ /** Validates vector size: must be a power of 2 in [2^MIN_LOG .. 2^MAX_LOG]. */
+ static int validateVectorSize(int vectorSize) {
+ Preconditions.checkArgument(
+ vectorSize > 0 && (vectorSize & (vectorSize - 1)) == 0,
Review Comment:
This is probably faster, but I think a simpler form would be easier to read and understand, like `Integer.bitCount(vectorSize) == 1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099486574
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
Review Comment:
It would be more clear to use "index" rather than "number" because indexes are 0-based.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099593710
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
Review Comment:
The term "compression body" is used a couple of times, but "body" isn't in the spec so it is not very well defined. I would remove it and use "after the header" or "from the start of the offset array" instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "prtkgaur (via GitHub)" <gi...@apache.org>.
prtkgaur commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-6150629419
cc @vinooganesh, can you please take a stab at review comments from Ryan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504072881
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int currentIndex;
+ protected int currentVectorIndex;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ AlpValuesReader() {
+ this.currentIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorIndex = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.currentIndex = 0;
+ this.currentVectorIndex = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
Review Comment:
slice(0) should work here for all nulls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504217087
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
+
+ // ---------------------------------------------------------------------------
+ // Helpers: build a known-good encoded page, then mutate copies of it
+ // ---------------------------------------------------------------------------
+
+ /** Build a valid ALP-encoded double page with N clean values. */
+ private static byte[] validDoublePage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(512, valueCount * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ // 2-decimal values — the ALP sweet spot, ensures no exceptions
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeDouble((i % 1000) / 100.0);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Build a valid ALP-encoded float page with N clean values. */
+ private static byte[] validFloatPage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(256, valueCount * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeFloat((i % 1000) / 100.0f);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Sanity baseline: the known-good page actually decodes cleanly. */
+ @Test
+ public void sanityBaselineDecodesClean() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Header-level validation (already-validated paths)
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsBadCompressionMode() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[0] = (byte) 0x99; // mode is at byte 0
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("compression"));
+ }
+
+ @Test
+ public void rejectsBadIntegerEncoding() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[1] = (byte) 0x99; // integer_encoding is at byte 1
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("integer encoding"));
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooLarge() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 99; // log_vector_size at byte 2
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("vector size"));
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooSmall() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 2; // below MIN_LOG_VECTOR_SIZE=3
+ assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ }
+
+ @Test
+ public void rejectsNegativeNumElements() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements is int32 LE at bytes 3..6 — write -1
+ page[3] = (byte) 0xFF;
+ page[4] = (byte) 0xFF;
+ page[5] = (byte) 0xFF;
+ page[6] = (byte) 0xFF;
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("element count"));
+ }
+
+ @Test
+ public void rejectsNumElementsGreaterThanValuesCount() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements stays 32; pass valuesCount=10 (smaller than encoded count)
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(10, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exceeds"));
+ }
+
+ // ---------------------------------------------------------------------------
+ // Vector-level validation (already-validated paths, surface lazily on decode)
+ // ---------------------------------------------------------------------------
+
+ /** Helper: find the byte position where the first vector's metadata starts. */
+ private static int firstVectorOffset(byte[] page) {
+ // header (7) + first 4 bytes of offset array = the offset value itself
+ int firstVectorOff = ByteBuffer.wrap(page, 7, 4).order(ByteOrder.LITTLE_ENDIAN).getInt();
+ // offsets are measured from the start of the compression body (after the 7B header)
+ return 7 + firstVectorOff;
+ }
+
+ @Test
+ public void rejectsExponentTooHighDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99; // exponent byte
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exponent"));
+ }
+
+ @Test
+ public void rejectsExponentTooHighFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exponent"));
+ }
+
+ @Test
+ public void rejectsFactorGreaterThanExponent() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 2; // exponent
+ page[v0 + 1] = (byte) 5; // factor > exponent
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("factor"));
+ }
+
+ @Test
+ public void rejectsTooManyExceptions() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // num_exceptions at v0+2, uint16 LE — set to 9999, way more than vectorLen=16
+ page[v0 + 2] = (byte) (9999 & 0xFF);
+ page[v0 + 3] = (byte) ((9999 >>> 8) & 0xFF);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("numexceptions"));
+ }
+
+ // ---------------------------------------------------------------------------
+ // Currently-unvalidated paths: truncation and corrupted offsets
+ // These currently fail with low-level Throwables (BufferUnderflowException,
+ // IndexOutOfBoundsException). The tests assert any Throwable is raised so we
+ // notice if a regression silently swallows the corruption.
+ // ---------------------------------------------------------------------------
+
+ /** Page with only the 7-byte header — nothing else. */
+ @Test
+ public void rejectsHeaderOnlyPage() {
+ byte[] tiny = new byte[]{0x00, 0x00, 0x0A, 0x20, 0x00, 0x00, 0x00}; // 32 elements, log_vec=10
Review Comment:
is this valid if all values in the page are null?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504142232
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
Review Comment:
same comment on constant, I think this is tied to the existing unpacking code?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3527032490
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpBitPackingTest.java:
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.junit.Test;
+
+/**
+ * Tests for bit-packing behavior in the ALP encoding pipeline.
+ */
+public class AlpBitPackingTest {
+
+ @Test
+ public void testBytePackerIntRoundTrip() {
+ // Verify BytePacker pack/unpack round-trip for various bit widths
+ for (int bitWidth = 1; bitWidth <= 31; bitWidth++) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int maxVal = (int) Math.min((1L << bitWidth) - 1, Integer.MAX_VALUE);
+
+ int[] input = new int[8];
+ for (int i = 0; i < 8; i++) {
+ input[i] = (maxVal / 8) * i;
+ }
+
+ byte[] packed = new byte[bitWidth];
+ packer.pack8Values(input, 0, packed, 0);
+
+ int[] unpacked = new int[8];
+ ByteBuffer buf = ByteBuffer.wrap(packed);
+ packer.unpack8Values(buf, 0, unpacked, 0);
+
+ for (int i = 0; i < 8; i++) {
+ assertEquals("BitWidth=" + bitWidth + " index=" + i, input[i], unpacked[i]);
+ }
+ }
+ }
+
+ @Test
+ public void testBytePackerForLongRoundTrip() {
+ // Verify BytePackerForLong pack/unpack round-trip for various bit widths
+ for (int bitWidth = 1; bitWidth <= 63; bitWidth++) {
Review Comment:
doesn't this need to test through 64?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3566513220
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
Review Comment:
Done. Pulled the group size out into PACK_GROUP_SIZE in AlpConstants and used it here (and in the float reader) instead of the bare 8.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
Review Comment:
Same here, using PACK_GROUP_SIZE now. And yeah, you're right that it's tied to the 8-wide unpack in the byte packer, so the constant just gives that a name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575348445
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ int bestExponent = presets[0][0];
Review Comment:
Added E and F index constants (0 and 1) and use them in the pair arrays.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ int bestExponent = presets[0][0];
+ int bestFactor = presets[0][1];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int[] preset : presets) {
+ int e = preset[0];
+ int f = preset[1];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
Review Comment:
Signed range here too, same fix as the unsigned question above: the delta is (long) maxEncoded - (long) minEncoded.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ int bestExponent = presets[0][0];
+ int bestFactor = presets[0][1];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int[] preset : presets) {
+ int e = preset[0];
+ int f = preset[1];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestDoubleParams(double[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= DOUBLE_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ long minEncoded = Long.MAX_VALUE;
+ long maxEncoded = Long.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ double value = values[offset + i];
+ if (isDoubleException(value, e, f)) {
+ exceptions++;
+ } else {
+ long encoded = encodeDouble(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ // delta as signed subtraction; Long.numberOfLeadingZeros handles the unsigned bit width
+ // correctly even when the subtraction overflows (large range → penalized with 64 bits).
+ long delta = (nonExceptions < 2) ? 0 : (maxEncoded - minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Double.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
Review Comment:
Collapsed into one pickBestFloat method. findBestFloatParams and findBestFloatParamsWithPresets both delegate to it, just with different pair lists (all pairs vs the cached presets).
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
Review Comment:
The writer uses BytesUtils.getWidthFromMaxInt where the delta fits in an int. The estimator computes the FOR span as a long (the signed range can exceed int), so it uses a long-width computation there, which is the bitWidthForLong helper, documented as the long counterpart since BytesUtils only has the int version.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
Review Comment:
Reordered so the sampling and cache-building path comes first (cachedPresets == null) and the cached path is the else, matching the flow you sketched.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,635 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ private final List<float[]> rowgroupSamples;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.rowgroupSamples = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump = Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[8];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Use cached presets after the sampling phase, full search before
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets != null) {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && rowgroupSamples.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ rowgroupSamples.add(Arrays.copyOf(vectorBuffer, vectorLen));
+ }
+ }
+
+ vectorsProcessed++;
+ if (cachedPresets == null && rowgroupSamples.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+
+ int excIdx = 0;
+
+ // We need a valid encoded value to fill exception slots (placeholder).
+ // Any non-exception value works; it gets overwritten on decode.
Review Comment:
Rewrote the comment along the lines you suggested: the placeholder is the first non-exception encoding (0 when the whole vector is exceptions), which avoids introducing a spurious FOR minimum, and the reader overwrites those slots from the exception payload so it only affects FOR/bit width.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3962456171
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -468,6 +504,8 @@ private Builder() {
DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED
? ByteStreamSplitMode.FLOATING_POINT
: ByteStreamSplitMode.NONE);
+ alp = ColumnProperty.<AlpConfig>builder()
Review Comment:
Silently overriding byteStreamSplitEnabled
We probably want a slightly more complicated logic here
If (bytestream && ALP) fail (can't use both)
... elif(bytestream) {
bystreamEnabled ...
} elif( alp) {
alpEnabled
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3960571730
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConfig.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import java.util.Objects;
+
+/**
+ * Immutable per-column ALP encoding configuration: whether ALP is enabled and the vector size
+ * (number of values per encoded vector) to use. Bundled together so a column carries a single
+ * cohesive ALP setting rather than several independent properties.
+ */
+public final class AlpConfig {
+
+ /** Default values per encoded vector. */
+ public static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ /** ALP disabled, with the default vector size. */
+ public static final AlpConfig DISABLED = new AlpConfig(false, DEFAULT_VECTOR_SIZE);
+
+ private final boolean enabled;
+ private final int vectorSize;
+
+ /**
+ * @param enabled whether ALP encoding is enabled
+ * @param vectorSize values per encoded vector; must be a power of 2 in the supported range
+ * @throws IllegalArgumentException if {@code vectorSize} is not a supported vector size
+ */
+ public AlpConfig(boolean enabled, int vectorSize) {
Review Comment:
There probably shouldn't be an "enabled" as part of this config. Whether ALP is on or off is separate from how it is configured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099674094
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
Review Comment:
I expected the "body" to mean the vector's "Data Section" (using the term from the spec) but instead it looks like this decodes the packed integer values, since the exception positions and exception values are decoded below in this method. A better name would help here, but I would also like to see the logic of decoding broken down along the same lines as the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098752199
##########
parquet-column/src/main/java/org/apache/parquet/column/Encoding.java:
##########
@@ -147,6 +149,26 @@ public ValuesReader getValuesReader(ColumnDescriptor descriptor, ValuesType valu
}
},
+ /**
+ * ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE types.
+ * Works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference (FOR) encoding and bit-packing.
Review Comment:
This isn't the right place to document how ALP works. This is probably a generated comment that can/should be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099418611
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
Review Comment:
I think this should be `vectorsData.remaining()` and not `limit()` because the limit is a position in the buffer. This is used in the check below, `offset > bodyLimit - minVectorBytes`. `offset` is relative to the start of the buffer, which is `buffer.position()`, but `limit` is relative to the start of the backing bytes and not relative to `position`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3508960593
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForFloat() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new float[capacity];
+ this.deltasBuffer = new int[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Integer.SIZE]; // max bit width for int = 32 bytes
+ }
+
+ @Override
+ public float readFloat() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP float data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > FLOAT_MAX_EXPONENT) {
+ throw new ParquetDecodingException(
+ "Invalid ALP float exponent " + exponent + " in vector " + vectorIdx + ", max is " + FLOAT_MAX_EXPONENT);
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException(
+ "Invalid ALP float factor " + factor + " > exponent " + exponent + " in vector " + vectorIdx);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException(
+ "Invalid ALP numExceptions " + numExceptions + " > vectorLen " + vectorLen + " in vector " + vectorIdx);
+ }
+
+ int frameOfReference = getIntLE(vectorsData, pos);
+ int bitWidth = vectorsData.get(pos + 4) & 0xFF;
+ pos += FLOAT_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
Review Comment:
Should we check that the bitwidth is also within reasonable bounds? bitWidth within [0,32] or [0,64] respectively?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504232644
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
this also needs to be removed once released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3608666457
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1066 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {
+ "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
+ if (dir == null) dir = System.getenv("ALP_TEST_DATA_DIR");
+ if (dir != null && new File(dir).isDirectory()) return Paths.get(dir);
+ // Default: alp-test-data/ relative to project root (two levels up from target/test-classes)
+ java.nio.file.Path candidate =
+ Paths.get(System.getProperty("user.dir")).resolve("alp-test-data");
+ return candidate.toFile().isDirectory() ? candidate : null;
+ }
+
+ private java.nio.file.Path getSingleTestFile() {
+ String filePath = System.getProperty("ALP_TEST_FILE");
+ if (filePath == null) filePath = System.getenv("ALP_TEST_FILE");
+ if (filePath != null && new File(filePath).exists()) return Paths.get(filePath);
+ return null;
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ @Test
+ public void testReadSingleAlpFile() throws IOException {
+ java.nio.file.Path file = getSingleTestFile();
+ assumeTrue("ALP_TEST_FILE not set or file does not exist, skipping", file != null);
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected at least one row in " + file, rows.size() > 0);
+ LOG.info("testReadSingleAlpFile: read {} rows from {}", rows.size(), file.getFileName());
+ }
+
+ @Test
+ public void testReadCppDoubleFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_DOUBLE_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppDoubleFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ @Test
+ public void testReadCppFloatFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_FLOAT_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppFloatFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ /** Verify no NaN/Inf corruption in float/double columns across all test files. */
+ @Test
+ public void testNoCorruptionInCppFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ int nanCount = 0;
+ int infCount = 0;
+ int totalValues = 0;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < pages.getRowCount(); i++) {
+ Group row = recordReader.read();
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ try {
+ if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float v = row.getFloat(f, 0);
+ totalValues++;
+ if (Float.isNaN(v)) nanCount++;
+ if (Float.isInfinite(v)) infCount++;
+ } else if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double v = row.getDouble(f, 0);
+ totalValues++;
+ if (Double.isNaN(v)) nanCount++;
+ if (Double.isInfinite(v)) infCount++;
+ }
+ } catch (Exception ignored) {
+ }
+ }
+ }
+ }
+ }
+ LOG.info(
+ "{}: {} values, {} NaN, {} Inf", filename, totalValues, nanCount, infCount);
+ assertEquals("Unexpected NaN in " + filename, 0, nanCount);
+ assertEquals("Unexpected Inf in " + filename, 0, infCount);
+ }
+ }
+
+ /** Schema inspection: log encoding types used in each column. */
+ @Test
+ public void testLogSchemaAndEncodings() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ LOG.info("=== {} ===", filename);
+ LOG.info("Schema: {}", schema);
+ if (!footer.getBlocks().isEmpty()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col :
+ footer.getBlocks().get(0).getColumns()) {
+ LOG.info(
+ " column={} encodings={} compression={}",
+ col.getPath(),
+ col.getEncodings(),
+ col.getCodec());
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not read metadata for {}: {}", filename, e.getMessage());
+ }
+ }
+ // This test always passes — it's for inspection
+ assertTrue(true);
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { "
+ + "required double double_col; "
+ + "required float float_col; "
+ + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch for " + version, DOUBLE_VALUES.length, rows.size());
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i + " for " + version,
+ DOUBLE_VALUES[i],
+ rows.get(i).getDouble("double_col", 0),
+ 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i + " for " + version,
+ FLOAT_VALUES[i],
+ rows.get(i).getFloat("float_col", 0),
+ 0.0f);
+ }
+ LOG.info("writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}", version, rows.size(), outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(4096)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch at vectorSize=4096", rowCount, rows.size());
+ for (int i = 0; i < rowCount; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i, doubles[i], rows.get(i).getDouble("double_col", 0), 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i, floats[i], rows.get(i).getFloat("float_col", 0), 0.0f);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
+ * V2 page headers include the encoding value directly; ALP = 10 is supported via the build-time
+ * patch to the generated Encoding enum in parquet-format-structures.
+ */
+ @Test
+ public void testJavaWriteAlpV2Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_2_0);
+ }
+
+ /**
+ * Writes ALP-encoded V1 and V2 parquet files from Java and verifies that pyarrow (Python Arrow)
+ * can read them back correctly. Skipped if python3/pyarrow is not available in the environment.
+ *
+ * <p>This test addresses cross-language compatibility: Java writes, another implementation reads.
+ */
+ @Test
+ public void testJavaWrittenAlpFilesReadableByPyarrow() throws IOException, InterruptedException {
+ // Check python3 is available
+ Process check = new ProcessBuilder("python3", "-c", "import pyarrow.parquet")
+ .redirectErrorStream(true)
+ .start();
+ assumeTrue("python3/pyarrow not available, skipping cross-language test", check.waitFor() == 0);
+
+ for (WriterVersion version : new WriterVersion[] {WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0}) {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_xcompat_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ // Build a python snippet that reads the file and asserts row count and values
+ StringBuilder pyScript = new StringBuilder();
+ pyScript.append("import pyarrow.parquet as pq, sys\n");
+ pyScript.append("t = pq.read_table('").append(outPath.toAbsolutePath()).append("')\n");
+ pyScript.append("assert len(t) == ").append(DOUBLE_VALUES.length).append(", f'row count mismatch: {len(t)}'\n");
+ pyScript.append("dc = t.column('double_col').to_pylist()\n");
+ pyScript.append("fc = t.column('float_col').to_pylist()\n");
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ pyScript.append("assert abs(dc[").append(i).append("] - ").append(DOUBLE_VALUES[i])
+ .append(") < 1e-9, f'double mismatch at ").append(i).append(": {dc[").append(i).append("]}'\n");
+ pyScript.append("assert abs(fc[").append(i).append("] - ").append(FLOAT_VALUES[i])
+ .append(") < 1e-4, f'float mismatch at ").append(i).append(": {fc[").append(i).append("]}'\n");
+ }
+ pyScript.append("print('OK: " + version.name() + " " + DOUBLE_VALUES.length + " rows verified by pyarrow')\n");
+
+ Process proc = new ProcessBuilder("python3", "-c", pyScript.toString())
+ .redirectErrorStream(true)
+ .start();
+ String output = new String(proc.getInputStream().readAllBytes());
+ int exitCode = proc.waitFor();
+ LOG.info("pyarrow cross-compat [{}]: {}", version, output.trim());
+ // pyarrow may not yet support reading ALP (Arrow C++ PR #48345 in progress).
+ // Skip rather than fail so the test becomes a passing signal once pyarrow adds ALP support.
+ assumeTrue(
+ "pyarrow does not yet support reading ALP encoding (Arrow C++ PR #48345 pending): "
+ + output,
+ !output.contains("Unknown encoding type"));
+ assertEquals(
+ "pyarrow failed to read Java-written ALP file (" + version + "): " + output, 0, exitCode);
+ }
+ }
+
+ // TODO: Uncomment once parquet-testing PR #100 merges
+ /*
+ @Test
+ public void testReadAlpFromParquetTesting() throws IOException {
+ // InterOpTester will auto-download from parquet-testing
+ }
+ */
+
+ // ---------------------------------------------------------------------------
+ // Fixture generator: reads the C++-written ALP files from parquet-testing
+ // PR #100 and re-encodes each as Java ALP at two vector sizes (1024 and 4096).
+ // Outputs land in ALP_OUTPUT_DIR (default: ${user.dir}/alp-java-generated/)
+ // and are intended for submission back to parquet-testing PR #100 so the
+ // C++/Rust/Go readers can verify cross-language compatibility against Java
+ // writes at both vector sizes.
+ // ---------------------------------------------------------------------------
+
+ private static final String[] SOURCE_FILES = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ private static final int[] GENERATOR_VECTOR_SIZES = {1024, 4096};
+ private static final WriterVersion[] GENERATOR_PAGE_VERSIONS = {
+ WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0
+ };
+
+ /**
+ * Resolves the explicit output directory from {@code ALP_OUTPUT_DIR} (system property or env var)
+ * and creates it if missing. Returns {@code null} when the variable is unset — callers should
+ * {@code assumeTrue} on the result so the test skips cleanly rather than writing stray files
+ * into the module tree (which would fail the RAT license check on subsequent {@code mvn test}
+ * runs in the same module).
+ */
+ private java.nio.file.Path getOutputDir() throws IOException {
+ String dir = System.getProperty("ALP_OUTPUT_DIR");
+ if (dir == null) dir = System.getenv("ALP_OUTPUT_DIR");
+ if (dir == null) return null;
+ java.nio.file.Path target = Paths.get(dir);
+ java.nio.file.Files.createDirectories(target);
+ return target;
+ }
+
+ /**
+ * Read all rows of the given source file and return them as a list of value-only Group rows
+ * keyed by the source's schema. Used for both the copy and the verification round-trip.
+ */
+ private List<Group> readGroupsWithSchema(java.nio.file.Path source, MessageType[] outSchema) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(source))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ outSchema[0] = schema;
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ /**
+ * Generates Java-written ALP fixtures from the four PR #100 source datasets across two axes:
+ * page version (V1, V2) and vectorSize (1024, 4096). Each output is verified bit-exact against
+ * its source. Produces 16 files total:
+ * {@code alp_java_{spotify1,arade,float_spotify1,float_arade}_v{1,2}_vs{1024,4096}.parquet}.
+ *
+ * <p>Page version is orthogonal to ALP encoding (the page version difference lives in the
+ * parquet protocol layer, not in the ALP payload), but covering both axes makes the fixture
+ * set fully symmetric for cross-language compatibility — every reader can verify it handles
+ * Java-written ALP regardless of how the surrounding pages are framed.
+ *
+ * <p>To run: clone https://github.com/prtkgaur/parquet-testing/tree/alpFloatingPointDataset
+ * and point ALP_TEST_DATA_DIR at its {@code data/} directory. Set ALP_OUTPUT_DIR to choose
+ * where to write the generated fixtures; if unset, this test skips.
+ */
+ @Test
+ public void generateAlpFixturesAtMultipleVectorSizes() throws IOException {
+ java.nio.file.Path sourceDir = getTestDataDir();
+ assumeTrue("ALP source dir not found, skipping fixture generator", sourceDir != null);
+
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping fixture generator", outDir != null);
+ LOG.info("Generating ALP fixtures to {}", outDir);
+
+ int expectedFiles =
+ SOURCE_FILES.length * GENERATOR_PAGE_VERSIONS.length * GENERATOR_VECTOR_SIZES.length;
+ int generated = 0;
+ for (String sourceFile : SOURCE_FILES) {
+ java.nio.file.Path source = sourceDir.resolve(sourceFile);
+ assumeTrue("Source missing: " + source, source.toFile().exists());
+
+ MessageType[] schemaHolder = new MessageType[1];
+ List<Group> sourceRows = readGroupsWithSchema(source, schemaHolder);
+ MessageType schema = schemaHolder[0];
+
+ String stem = sourceFile.replace("alp_", "").replace(".parquet", "");
+
+ for (WriterVersion pageVersion : GENERATOR_PAGE_VERSIONS) {
+ String pageTag = (pageVersion == WriterVersion.PARQUET_1_0) ? "v1" : "v2";
+ for (int vectorSize : GENERATOR_VECTOR_SIZES) {
+ java.nio.file.Path outPath =
+ outDir.resolve(
+ "alp_java_" + stem + "_" + pageTag + "_vs" + vectorSize + ".parquet");
+ java.nio.file.Files.deleteIfExists(outPath);
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(pageVersion)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(vectorSize)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (Group row : sourceRows) {
+ writer.write(row);
+ }
+ }
+
+ // Verify: read back and compare against the source row-by-row
+ List<Group> roundTrip = readGroupsWithSchema(outPath, new MessageType[1]);
+ assertEquals(
+ "Row count mismatch for " + outPath.getFileName(),
+ sourceRows.size(),
+ roundTrip.size());
+ int fieldCount = schema.getFieldCount();
+ for (int i = 0; i < sourceRows.size(); i++) {
+ for (int f = 0; f < fieldCount; f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ String fieldName = schema.getType(f).getName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double expected = sourceRows.get(i).getDouble(f, 0);
+ double actual = roundTrip.get(i).getDouble(f, 0);
+ long eb = Double.doubleToRawLongBits(expected);
+ long ab = Double.doubleToRawLongBits(actual);
+ assertEquals(
+ "double bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float expected = sourceRows.get(i).getFloat(f, 0);
+ float actual = roundTrip.get(i).getFloat(f, 0);
+ int eb = Float.floatToRawIntBits(expected);
+ int ab = Float.floatToRawIntBits(actual);
+ assertEquals(
+ "float bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ }
+ }
+ }
+ long bytes = outPath.toFile().length();
+ LOG.info(
+ " wrote {} ({} rows, {} bytes, pageVersion={}, vectorSize={})",
+ outPath.getFileName(),
+ sourceRows.size(),
+ bytes,
+ pageTag,
+ vectorSize);
+ generated++;
+ }
+ }
+ }
+ assertEquals(
+ "Expected to generate " + expectedFiles + " fixture files", expectedFiles, generated);
+ LOG.info("Generated {} ALP fixture files to {}", generated, outDir);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Reader-only validation: opens each Java-written fixture and asserts the
+ // reader path works end-to-end (metadata declares ALP, every row decodes
+ // without error). Separate from the generator's own round-trip check so it
+ // surfaces as a distinct signal — useful when the fixtures already exist on
+ // disk and you want to validate just the read path.
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void readAllFixtureFilesIndependently() throws IOException {
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping reader-only test", outDir != null);
+ File[] files =
+ outDir.toFile().listFiles((f, n) -> n.startsWith("alp_java_") && n.endsWith(".parquet"));
+ assumeTrue("No fixture files in " + outDir, files != null && files.length > 0);
+ java.util.Arrays.sort(files, (a, b) -> a.getName().compareTo(b.getName()));
+
+ int filesRead = 0;
+ int totalChunks = 0;
+ int alpChunks = 0;
+ long totalRows = 0;
+ for (File pf : files) {
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(pf.toPath()))) {
+ ParquetMetadata footer = reader.getFooter();
+ // Layer 1: every column chunk declares ALP
+ for (org.apache.parquet.hadoop.metadata.BlockMetaData block : footer.getBlocks()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col : block.getColumns()) {
+ totalChunks++;
+ if (col.getEncodings().contains(Encoding.ALP)) alpChunks++;
+ }
+ }
+ // Layer 2: every row actually decodes (no exceptions thrown, count matches footer)
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ long rowsDecoded = 0;
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ Group row = recordReader.read();
+ // Touch every present field so any decode-time exception surfaces.
+ // Optional/null fields have repetitionCount=0 — skip those rather than throw.
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ if (row.getFieldRepetitionCount(f) == 0) continue;
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ row.getDouble(f, 0);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ row.getFloat(f, 0);
+ }
+ }
+ rowsDecoded++;
+ }
+ }
+ totalRows += rowsDecoded;
+ filesRead++;
+ }
+ }
+ assertEquals("Every column chunk should declare ALP", totalChunks, alpChunks);
+ LOG.info(
+ "readAllFixtureFilesIndependently: {} files, {} chunks ({} ALP), {} rows decoded",
+ filesRead, totalChunks, alpChunks, totalRows);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Corner-case fixture per parquet-testing issue #105. Writes a single small
+ // file with one ALP-encoded column per corner case (no exceptions, 1 exception,
+ // all exceptions, NaN/Inf/-0.0, nulls, constant/bit_width=0, multi-vector with
+ // differing exponents) for both f32 and f64, then reads back and verifies
+ // every value bit-exactly against the expected pattern.
+ //
+ // The output file (alp_java_cornercases.parquet) is intended as a candidate
+ // fixture for parquet-testing PR #100 once Prateek confirms naming and design.
+ // ---------------------------------------------------------------------------
+
+ private static final int CORNER_ROWS_PER_VECTOR = 1024;
+ private static final int CORNER_VECTORS = 2;
+ private static final int CORNER_TOTAL_ROWS = CORNER_ROWS_PER_VECTOR * CORNER_VECTORS;
+
+ /** Expected values per corner-case column. Doubles use NaN to mark a null cell for sentinel use. */
+ private static class CornerCaseData {
+ double[] doubleVals;
+ float[] floatVals;
+ boolean[] isNull; // only meaningful for optional columns
+ boolean isFloat;
+ boolean isOptional;
+ String name;
+
+ static CornerCaseData doubles(String name, double[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.doubleVals = vals;
+ d.isFloat = false;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData floats(String name, float[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.floatVals = vals;
+ d.isFloat = true;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData optionalDoubles(String name, double[] vals, boolean[] nulls) {
+ CornerCaseData d = doubles(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+
+ static CornerCaseData optionalFloats(String name, float[] vals, boolean[] nulls) {
+ CornerCaseData d = floats(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+ }
+
+ private List<CornerCaseData> buildCornerCaseColumns() {
+ List<CornerCaseData> cols = new ArrayList<>();
+ int N = CORNER_TOTAL_ROWS;
+
+ // 1. f64_no_exceptions: clean 2-decimal values, both vectors low bit-width
+ double[] noExc = new double[N];
+ for (int i = 0; i < N; i++) noExc[i] = (i % 1000) / 100.0;
+ cols.add(CornerCaseData.doubles("f64_no_exceptions", noExc));
+
+ // 2. f64_one_exception_per_vector: clean values with 1 NaN per 1024-row vector
+ double[] oneExc = new double[N];
+ for (int i = 0; i < N; i++) oneExc[i] = (i % 1000) / 100.0;
+ oneExc[100] = Double.NaN;
+ oneExc[1100] = Double.NaN;
+ cols.add(CornerCaseData.doubles("f64_one_exception_per_vector", oneExc));
+
+ // 3. f64_all_nan: every value NaN — all-exception vector
+ double[] allNan = new double[N];
+ java.util.Arrays.fill(allNan, Double.NaN);
+ cols.add(CornerCaseData.doubles("f64_all_nan", allNan));
+
+ // 4. f64_nan_inf_negzero: mix of special values across two vectors
+ double[] mixed = new double[N];
+ for (int i = 0; i < N; i++) mixed[i] = (i % 100) / 10.0;
+ mixed[0] = Double.NaN;
+ mixed[1] = Double.POSITIVE_INFINITY;
+ mixed[2] = Double.NEGATIVE_INFINITY;
+ mixed[3] = -0.0;
+ mixed[1024] = Double.NaN;
+ mixed[1025] = -0.0;
+ cols.add(CornerCaseData.doubles("f64_nan_inf_negzero", mixed));
+
+ // 5. f64_constant: every value = 42.0 → max_delta=0 → bit_width=0
+ double[] constant = new double[N];
+ java.util.Arrays.fill(constant, 42.0);
+ cols.add(CornerCaseData.doubles("f64_constant", constant));
+
+ // 6. f64_different_e_f_per_vector: vector 0 uses e=2,f=0 (cents); vector 1 uses e=3,f=0 (mils)
+ double[] diff = new double[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diff[i] = (i % 100) / 100.0;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diff[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0;
+ cols.add(CornerCaseData.doubles("f64_different_ef_per_vector", diff));
+
+ // 7. f64_with_nulls: optional column, ~30% nulls scattered
+ double[] withNulls = new double[N];
+ boolean[] nullMask = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMask[i] = true;
+ } else {
+ withNulls[i] = (i % 100) / 100.0;
+ }
+ }
+ cols.add(CornerCaseData.optionalDoubles("f64_with_nulls", withNulls, nullMask));
+
+ // 8–14: f32 analogues
+ float[] noExcF = new float[N];
+ for (int i = 0; i < N; i++) noExcF[i] = (i % 1000) / 100.0f;
+ cols.add(CornerCaseData.floats("f32_no_exceptions", noExcF));
+
+ float[] oneExcF = new float[N];
+ for (int i = 0; i < N; i++) oneExcF[i] = (i % 1000) / 100.0f;
+ oneExcF[100] = Float.NaN;
+ oneExcF[1100] = Float.NaN;
+ cols.add(CornerCaseData.floats("f32_one_exception_per_vector", oneExcF));
+
+ float[] allNanF = new float[N];
+ java.util.Arrays.fill(allNanF, Float.NaN);
+ cols.add(CornerCaseData.floats("f32_all_nan", allNanF));
+
+ float[] mixedF = new float[N];
+ for (int i = 0; i < N; i++) mixedF[i] = (i % 100) / 10.0f;
+ mixedF[0] = Float.NaN;
+ mixedF[1] = Float.POSITIVE_INFINITY;
+ mixedF[2] = Float.NEGATIVE_INFINITY;
+ mixedF[3] = -0.0f;
+ mixedF[1024] = Float.NaN;
+ mixedF[1025] = -0.0f;
+ cols.add(CornerCaseData.floats("f32_nan_inf_negzero", mixedF));
+
+ float[] constantF = new float[N];
+ java.util.Arrays.fill(constantF, 42.0f);
+ cols.add(CornerCaseData.floats("f32_constant", constantF));
+
+ float[] diffF = new float[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diffF[i] = (i % 100) / 100.0f;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diffF[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0f;
+ cols.add(CornerCaseData.floats("f32_different_ef_per_vector", diffF));
+
+ float[] withNullsF = new float[N];
+ boolean[] nullMaskF = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMaskF[i] = true;
+ } else {
+ withNullsF[i] = (i % 100) / 100.0f;
+ }
+ }
+ cols.add(CornerCaseData.optionalFloats("f32_with_nulls", withNullsF, nullMaskF));
+
+ return cols;
+ }
+
+ /**
+ * Dumps the corner-case construction recipe to a CSV truth file alongside the parquet.
+ * The CSV is built directly from the expected values (not by reading the parquet back),
+ * which makes it suitable as independent ground truth for cross-language verifiers.
+ *
+ * <p>Conventions:
+ * <ul>
+ * <li>Header row matches column names.
+ * <li>Empty field marks a null cell in an optional column.
+ * <li>Special values use Java's standard toString output: {@code NaN}, {@code Infinity},
+ * {@code -Infinity}. These parse correctly via C++ {@code std::stod} / {@code std::stof}
+ * (per the standard, both are case-insensitive and accept the full word "Infinity").
+ * <li>{@code -0.0} round-trips through both Java and C++ string conversion bit-exact.
+ * </ul>
+ */
+ private void writeCornerCaseCsvTruth(java.nio.file.Path csvPath, List<CornerCaseData> cols) throws IOException {
+ try (java.io.BufferedWriter w = java.nio.file.Files.newBufferedWriter(csvPath)) {
+ // Header
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ w.write(cols.get(i).name);
+ }
+ w.write("\n");
+ // Data
+ for (int r = 0; r < CORNER_TOTAL_ROWS; r++) {
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ CornerCaseData c = cols.get(i);
+ if (c.isOptional && c.isNull[r]) {
+ // empty field = null
+ } else if (c.isFloat) {
+ w.write(Float.toString(c.floatVals[r]));
+ } else {
+ w.write(Double.toString(c.doubleVals[r]));
+ }
+ }
+ w.write("\n");
+ }
+ }
+ }
+
+ @Test
+ public void generateAndVerifyCornerCaseFixture() throws IOException {
Review Comment:
Correction to my caveat above: it was wrong. Extreme values do encode losslessly at the full FOR width, so I added direct coverage for exactly that:
- f64_extreme_for_64bit: exact integer-valued doubles near +/- 2^63. Scale of 1, zero exceptions, and the FOR delta overflows the signed max - min subtraction, forcing 64-bit packing (the reader reconstructs via 2's-complement wraparound).
- f64_extreme_for_63bit: doubles near +/- 2^61, giving 63-bit FOR without overflow (the non-overflow counterpart).
- f32_extreme_for_32bit: floats near +/- 2^31, 32-bit FOR, zero exceptions.
Each has an always-run unit test (testDoubleExtremeForBitWidth / testDoubleExtremeFor63BitWidth / testFloatExtremeForBitWidth) plus a corner-case fixture column, so they join the cross-language fixture set for the interop check. My earlier claim that these turn into exceptions only holds for the literal INT/LONG max value itself; values right up to the encoding limit encode fine and exercise the full bit width.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099349434
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
Review Comment:
I would recommend checking this above instead of the length check to gate allocating and decoding the offsets. That would then simplify this loop, which can be from 1 to numVectors instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099561316
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
Review Comment:
I found this a little confusing at first. In this case, "length" means the number of values in the vector, which we call "size" elsewhere (i.e. `vectorSize`). What makes it worse is that this is between two methods that return buffer positions for vectors based on vector number, so I was thinking in terms of buffer position and size.
I think this should be moved to keep the two position methods together, and also renamed to `vectorSize(int vectorIndex)`. Javadoc would also help. Also consider making this static and passing in the vector size and total value count. That would make this a lot more obvious:
```java
int position = getVectorDataPosition(currentIndex);
int length = vectorLength(currentIndex); // UNCLEAR WHAT THIS IS
int position = getVectorDataPosition(currentIndex);
int size = getVectorSize(currentIndex, valuesPerVector, totalValues); // MORE OBVIOUSLY VALUE COUNT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099583430
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
Review Comment:
I find static helpers can improve readability in cases like this. If you instead used `isLast(vectorIndex)` then it would be easier to understand. `isLast` should get inlined automatically and you can reuse it in several methods.
```java
if (isLast(vectorIndex)) {
int lastVectorSize = totalCount % valuesPerVector;
return lastVectorSize == 0 ? vectorSize : lastVectorSize;
}
return vectorSize;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098789538
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
Review Comment:
Not the place to link to the paper or explain what ALP is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098904963
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
Review Comment:
`currentVectorIndex`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099268752
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
Review Comment:
This doesn't seem to be a very effective bound. Using the default 1024 value vector and assuming the bad num_elements is as high as it can be, 2^31-1, this lower bound is about 124k (2 billion / 1024 / 17), which is about 1/8 of a 1MB page.
I think a better check is to validate the offset array. The spec says:
> Each offset gives the byte position of the corresponding vector's data, *measured from the start of the offset array itself.*
That means the first offset must be `4 * numVectors`, which is an easy check that you aren't reading bad data. If that has the expected value, then you can allocate the offsets array. There are more steps beyond this that would also be reasonable, like validating the first 1024 offsets (incrementing by at least 13 or 17 bytes, and no more than `17 + 8 * vectorSize` or `13 + 4 * vectorSize`). I probably would not go beyond the first check unless we start seeing issues though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099505538
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
Review Comment:
This is mutable. I'd prefer it if this used `vectorOffsets[numVectors + 1]` so that `vectorIndex + 1` is always valid. Then this check could throw `IllegalArgumentException` when `vectorIndex >= numVectors`.
I think that would be safer anyway. Currently, this will return `vectorsData.limit()` when called with `Integer.MAX_VALUE`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub) via issues" <is...@parquet.apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-6594758052
@wgtmac Thanks, merged master in. The earlier failure was master's TestFooterIntegrity, which #3878 fixed, so CI should be green once it runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-4846802780
@wgtmac @Fokko would you have a chance to take a look at this to make sure it looks reasonable, the C++ implementation is an pretty reasonable state, having a committer review this I think unblocks the vote on ALP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3509149308
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpValuesEndToEndTest.java:
##########
@@ -0,0 +1,1745 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Random;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * End-to-end tests for ALP encoding and decoding pipeline.
+ */
+public class AlpValuesEndToEndTest {
+
+ private static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ // ========== Helper Methods ==========
+
+ private void roundTripFloat(float[] values) throws Exception {
+ roundTripFloat(values, DEFAULT_VECTOR_SIZE);
+ }
+
+ private void roundTripFloat(float[] values, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(256, values.length * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), vectorSize);
+
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float expected = values[i];
+ float actual = reader.readFloat();
+
+ if (Float.isNaN(expected)) {
+ assertTrue("Expected NaN at index " + i, Float.isNaN(actual));
+ } else {
+ assertEquals(
+ "Value mismatch at index " + i + " for " + expected,
+ Float.floatToRawIntBits(expected),
+ Float.floatToRawIntBits(actual));
+ }
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ private void roundTripDouble(double[] values) throws Exception {
+ roundTripDouble(values, DEFAULT_VECTOR_SIZE);
+ }
+
+ private void roundTripDouble(double[] values, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(512, values.length * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), vectorSize);
+
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ double expected = values[i];
+ double actual = reader.readDouble();
+
+ if (Double.isNaN(expected)) {
+ assertTrue("Expected NaN at index " + i, Double.isNaN(actual));
+ } else {
+ assertEquals(
+ "Value mismatch at index " + i + " for " + expected,
+ Double.doubleToRawLongBits(expected),
+ Double.doubleToRawLongBits(actual));
+ }
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Float Pipeline Tests ==========
+
+ @Test
+ public void testFloatSingleValue() throws Exception {
+ roundTripFloat(new float[] {1.23f});
+ }
+
+ @Test
+ public void testFloatSmallBatch() throws Exception {
+ roundTripFloat(new float[] {0.0f, 1.0f, -1.0f, 3.14f, 100.5f, 0.001f, 1234567.0f});
+ }
+
+ @Test
+ public void testFloatRandomData() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 10000.0f - 5000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatWithExceptions() throws Exception {
+ roundTripFloat(new float[] {
+ 1.0f, Float.NaN, 2.0f, Float.POSITIVE_INFINITY, 3.0f, Float.NEGATIVE_INFINITY, 4.0f, -0.0f, 5.0f
+ });
+ }
+
+ @Test
+ public void testFloatAllExceptions() throws Exception {
+ roundTripFloat(new float[] {Float.NaN, Float.POSITIVE_INFINITY, Float.NEGATIVE_INFINITY, -0.0f});
+ }
+
+ @Test
+ public void testFloatMultipleVectors() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[3000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 1000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatZeroValues() throws Exception {
+ roundTripFloat(new float[] {0.0f, 0.0f, 0.0f, 0.0f});
+ }
+
+ @Test
+ public void testFloatIdenticalValues() throws Exception {
+ float[] values = new float[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 3.14159f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Vector Size / Boundary Tests ==========
+
+ @Test
+ public void testFloatExactOneVector() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatExactTwoVectors() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 2];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatExactThreeVectors() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 3];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatPartialLastVectorPlusOne() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE + 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatPartialLastVectorMinusOne() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 2 - 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatCustomVectorSize512() throws Exception {
+ float[] values = new float[512 + 100]; // partial second vector
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1f;
+ }
+ roundTripFloat(values, 512);
+ }
+
+ @Test
+ public void testFloatCustomVectorSize2048() throws Exception {
+ float[] values = new float[2048 * 2 + 500]; // partial third vector
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1f;
+ }
+ roundTripFloat(values, 2048);
+ }
+
+ @Test
+ public void testFloatCustomVectorSizeSmall() throws Exception {
+ // Minimum vector size = 8 (2^3)
+ float[] values = new float[25]; // 3 full vectors + 1 partial
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 1.5f;
+ }
+ roundTripFloat(values, 8);
+ }
+
+ // ========== Float: Large Page ==========
+
+ @Test
+ public void testFloatLargePage() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[15000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 100000.0f - 50000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Monetary Data ==========
+
+ @Test
+ public void testFloatMonetaryData() throws Exception {
+ float[] values = {
+ 19.99f, 29.95f, 9.99f, 49.99f, 99.95f, 14.50f, 7.99f, 24.99f, 39.95f, 59.99f, 119.99f, 4.99f, 0.99f, 1.50f,
+ 2.99f, 3.49f
+ };
+ roundTripFloat(values);
+ }
+
+ // ========== Float: All Identical (bit_width = 0) ==========
+
+ @Test
+ public void testFloatAllIdenticalBitWidthZero() throws Exception {
+ float[] values = new float[2000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 42.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Mixed Exceptions at Various Positions ==========
+
+ @Test
+ public void testFloatExceptionsAtBoundaries() throws Exception {
+ // Exceptions at vector start and end
+ float[] values = new float[20];
+ values[0] = Float.NaN;
+ for (int i = 1; i < values.length - 1; i++) {
+ values[i] = i * 1.1f;
+ }
+ values[values.length - 1] = Float.POSITIVE_INFINITY;
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Skip Tests ==========
+
+ @Test
+ public void testFloatSkip() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+
+ float[] values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, 9.0f, 10.0f};
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ reader.skip(3);
+ assertEquals(Float.floatToRawIntBits(5.0f), Float.floatToRawIntBits(reader.readFloat()));
+ reader.skip(2);
+ assertEquals(Float.floatToRawIntBits(8.0f), Float.floatToRawIntBits(reader.readFloat()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test
+ public void testFloatSkipAcrossVectorBoundaries() throws Exception {
+ int vectorSize = 8; // small vector for boundary testing
+ int totalValues = vectorSize * 3 + 4; // 3 full vectors + partial
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ totalValues * 8, totalValues * 8, new DirectByteBufferAllocator(), vectorSize);
+
+ float[] values = new float[totalValues];
+ for (int i = 0; i < totalValues; i++) {
+ values[i] = (i + 1) * 1.0f;
+ writer.writeFloat(values[i]);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(totalValues, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ // Read first value from vector 0
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+
+ // Skip past vector 0 and vector 1, into vector 2
+ reader.skip(vectorSize * 2 - 1); // skip to index vectorSize*2
+
+ // Read first value of vector 2
+ float expected = (vectorSize * 2 + 1) * 1.0f;
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+
+ // Skip into the partial last vector
+ reader.skip(vectorSize - 1); // skip to index vectorSize*3
+
+ expected = (vectorSize * 3 + 1) * 1.0f;
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Float: Writer Reset ==========
+
+ @Test
+ public void testFloatWriterReset() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+ writer.reset();
+ assertEquals(0, writer.getBufferedSize());
+
+ float[] values = {3.0f, 4.0f, 5.0f};
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (float expected : values) {
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Double Pipeline Tests ==========
+
+ @Test
+ public void testDoubleSingleValue() throws Exception {
+ roundTripDouble(new double[] {1.23456789});
+ }
+
+ @Test
+ public void testDoubleSmallBatch() throws Exception {
+ roundTripDouble(new double[] {0.0, 1.0, -1.0, 3.14159265358979, 100.5, 0.001, 12345678901234.0});
+ }
+
+ @Test
+ public void testDoubleRandomData() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 1000000.0 - 500000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleWithExceptions() throws Exception {
+ roundTripDouble(new double[] {
+ 1.0, Double.NaN, 2.0, Double.POSITIVE_INFINITY, 3.0, Double.NEGATIVE_INFINITY, 4.0, -0.0, 5.0
+ });
+ }
+
+ @Test
+ public void testDoubleAllExceptions() throws Exception {
+ roundTripDouble(new double[] {Double.NaN, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY, -0.0});
+ }
+
+ @Test
+ public void testDoubleMultipleVectors() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[3000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 100000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleZeroValues() throws Exception {
+ roundTripDouble(new double[] {0.0, 0.0, 0.0, 0.0});
+ }
+
+ @Test
+ public void testDoubleIdenticalValues() throws Exception {
+ double[] values = new double[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 3.14159265358979;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Vector Size / Boundary Tests ==========
+
+ @Test
+ public void testDoubleExactOneVector() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleExactTwoVectors() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE * 2];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoublePartialLastVectorPlusOne() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE + 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoublePartialLastVectorMinusOne() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE * 2 - 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSize512() throws Exception {
+ double[] values = new double[512 + 100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1;
+ }
+ roundTripDouble(values, 512);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSize2048() throws Exception {
+ double[] values = new double[2048 * 2 + 500];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1;
+ }
+ roundTripDouble(values, 2048);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSizeSmall() throws Exception {
+ double[] values = new double[25];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 1.5;
+ }
+ roundTripDouble(values, 8);
+ }
+
+ // ========== Double: Large Page ==========
+
+ @Test
+ public void testDoubleLargePage() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[15000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 100000.0 - 50000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Monetary Data ==========
+
+ @Test
+ public void testDoubleMonetaryData() throws Exception {
+ double[] values = {
+ 19.99, 29.95, 9.99, 49.99, 99.95, 14.50, 7.99, 24.99, 39.95, 59.99, 119.99, 4.99, 0.99, 1.50, 2.99, 3.49
+ };
+ roundTripDouble(values);
+ }
+
+ // ========== Double: All Identical (bit_width = 0) ==========
+
+ @Test
+ public void testDoubleAllIdenticalBitWidthZero() throws Exception {
+ double[] values = new double[2000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 42.0;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Mixed Exceptions at Boundaries ==========
+
+ @Test
+ public void testDoubleExceptionsAtBoundaries() throws Exception {
+ double[] values = new double[20];
+ values[0] = Double.NaN;
+ for (int i = 1; i < values.length - 1; i++) {
+ values[i] = i * 1.1;
+ }
+ values[values.length - 1] = Double.POSITIVE_INFINITY;
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Skip Tests ==========
+
+ @Test
+ public void testDoubleSkip() throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+
+ double[] values = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0};
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Double.doubleToRawLongBits(1.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(3);
+ assertEquals(Double.doubleToRawLongBits(5.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(2);
+ assertEquals(Double.doubleToRawLongBits(8.0), Double.doubleToRawLongBits(reader.readDouble()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test
+ public void testDoubleSkipAcrossVectorBoundaries() throws Exception {
+ int vectorSize = 8;
+ int totalValues = vectorSize * 3 + 4;
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ totalValues * 16, totalValues * 16, new DirectByteBufferAllocator(), vectorSize);
+
+ double[] values = new double[totalValues];
+ for (int i = 0; i < totalValues; i++) {
+ values[i] = (i + 1) * 1.0;
+ writer.writeDouble(values[i]);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(totalValues, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Double.doubleToRawLongBits(1.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(vectorSize * 2 - 1);
+
+ double expected = (vectorSize * 2 + 1) * 1.0;
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+
+ reader.skip(vectorSize - 1);
+ expected = (vectorSize * 3 + 1) * 1.0;
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Double: Writer Reset ==========
+
+ @Test
+ public void testDoubleWriterReset() throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+
+ writer.writeDouble(1.0);
+ writer.writeDouble(2.0);
+ writer.reset();
+ assertEquals(0, writer.getBufferedSize());
+
+ double[] values = {3.0, 4.0, 5.0};
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (double expected : values) {
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Packed Size Verification ==========
+
+ @Test
+ public void testPackedSizeSpecFormula() throws Exception {
+ // Verify that the packed data size exactly matches ceil(n * bitWidth / 8)
+ // by checking total encoded bytes for partial vectors with various remainders.
+ // Vector size 8 means: 1 full group of 8 = no partial group,
+ // but with count not divisible by 8 we exercise partial groups.
+ for (int count : new int[] {1, 2, 3, 5, 7, 9, 11, 13, 15}) {
+ float[] values = new float[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 1.0f;
+ }
+ // Use smallest vector size so the partial vector IS the only vector
+ roundTripFloat(values, 1024);
+ }
+ }
+
+ @Test
+ public void testPartialVectorAllRemaindersMod8() throws Exception {
+ // Test every possible remainder (1..7) to exercise the partial group path.
+ // With vectorSize=16: 16 is 2 full groups of 8, no partial group.
+ // So use vectorSize=16 and write 16+R values where R in {1..7}
+ // to get a partial last vector of size R.
+ for (int remainder = 1; remainder <= 7; remainder++) {
+ int count = 16 + remainder;
+ float[] values = new float[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 0.5f;
+ }
+ roundTripFloat(values, 16);
+ }
+ // Also test for doubles
+ for (int remainder = 1; remainder <= 7; remainder++) {
+ int count = 16 + remainder;
+ double[] values = new double[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 0.5;
+ }
+ roundTripDouble(values, 16);
+ }
+ }
+
+ // ========== Negative Encoded Values / Wide FOR Range ==========
+
+ @Test
+ public void testFloatNegativeValues() throws Exception {
+ // Values that produce negative encoded integers, testing FOR with negative min
+ float[] values = new float[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = -50.0f + i * 1.0f; // range [-50, 49]
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleNegativeValues() throws Exception {
+ double[] values = new double[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = -50.0 + i * 1.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatWideRange() throws Exception {
+ // Mix of small and large positive values - exercises higher bit widths
+ float[] values = {0.01f, 0.02f, 1000.0f, 2000.0f, 0.05f, 50000.0f, 0.99f, 99999.0f};
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleWideRange() throws Exception {
+ double[] values = {0.01, 0.02, 100000.0, 200000.0, 0.05, 5000000.0, 0.99, 99999999.0};
+ roundTripDouble(values);
+ }
+
+ // ========== Binary Format Verification ==========
+
+ @Test
+ public void testBinaryFormatExactBytes() throws Exception {
+ // Write a known small dataset with vectorSize=8, then verify the
+ // exact binary layout byte-by-byte
+ int vectorSize = 8;
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), vectorSize);
+
+ // Write 10 identical values: 1.0f
+ // With e=0, f=0: encoded=1, FOR min=1, all deltas=0, bitWidth=0
+ for (int i = 0; i < 10; i++) {
+ writer.writeFloat(1.0f);
+ }
+
+ BytesInput input = writer.getBytes();
+ byte[] bytes = input.toByteArray();
+ java.nio.ByteBuffer buf = java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ assertEquals("compression_mode", 0, buf.get() & 0xFF);
+ assertEquals("integer_encoding", 0, buf.get() & 0xFF);
+ assertEquals("log_vector_size", 3, buf.get() & 0xFF); // log2(8)=3
+ assertEquals("num_elements", 10, buf.getInt());
+
+ // Offset array (2 vectors * 4 bytes = 8 bytes)
+ int offset0 = buf.getInt(); // first vector offset
+ int offset1 = buf.getInt(); // second vector offset
+ assertEquals("offset0 = past offset array", 8, offset0);
+
+ // Vector 0: 8 identical values, bitWidth=0, no exceptions
+ // AlpInfo (4 bytes)
+ int v0Exp = buf.get() & 0xFF;
+ int v0Fac = buf.get() & 0xFF;
+ int v0Exc = buf.getShort() & 0xFFFF;
+ assertEquals("v0 exceptions", 0, v0Exc);
+
+ // ForInfo (5 bytes)
+ int v0For = buf.getInt();
+ int v0Bw = buf.get() & 0xFF;
+ assertEquals("v0 bitWidth", 0, v0Bw);
+ // No packed bytes (bitWidth=0)
+ // No exceptions
+
+ // Vector 0 should be exactly 4 + 5 = 9 bytes
+ assertEquals("offset1 should be offset0 + 9", offset0 + 9, offset1);
+
+ // Vector 1: 2 identical values, bitWidth=0, no exceptions
+ int v1Exp = buf.get() & 0xFF;
+ int v1Fac = buf.get() & 0xFF;
+ assertEquals("same exponent both vectors", v0Exp, v1Exp);
+ assertEquals("same factor both vectors", v0Fac, v1Fac);
+ int v1Exc = buf.getShort() & 0xFFFF;
+ assertEquals("v1 exceptions", 0, v1Exc);
+ int v1For = buf.getInt();
+ int v1Bw = buf.get() & 0xFF;
+ assertEquals("v1 bitWidth", 0, v1Bw);
+
+ // Should have consumed all bytes
+ assertEquals("all bytes consumed", bytes.length, buf.position());
+
+ // Verify round-trip
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(10, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < 10; i++) {
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Exception-Heavy Vectors ==========
+
+ @Test
+ public void testFloatManyExceptionsInVector() throws Exception {
+ // Vector with more exceptions than normal values
+ float[] values = new float[10];
+ values[0] = 1.0f; // normal
+ for (int i = 1; i < 10; i++) {
+ values[i] = Float.NaN; // all exceptions except first
+ }
+ roundTripFloat(values, 16);
+ }
+
+ @Test
+ public void testDoubleManyExceptionsInVector() throws Exception {
+ double[] values = new double[10];
+ values[0] = 1.0;
+ for (int i = 1; i < 10; i++) {
+ values[i] = Double.NaN;
+ }
+ roundTripDouble(values, 16);
+ }
+
+ @Test
+ public void testFloatExceptionsAcrossMultipleVectors() throws Exception {
+ // Exceptions in different vectors
+ int vectorSize = 8;
+ float[] values = new float[vectorSize * 3];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (i + 1) * 0.5f;
+ }
+ // Place exceptions at specific positions in different vectors
+ values[0] = Float.NaN; // vector 0, position 0
+ values[vectorSize - 1] = Float.POSITIVE_INFINITY; // vector 0, last position
+ values[vectorSize] = -0.0f; // vector 1, position 0
+ values[vectorSize * 2 + 3] = Float.NaN; // vector 2, position 3
+ roundTripFloat(values, vectorSize);
+ }
+
+ // ========== Read After Skip to End ==========
+
+ @Test(expected = ParquetDecodingException.class)
+ public void testFloatReadPastEnd() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ writer.writeFloat(1.0f);
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(1, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ reader.readFloat(); // read the one value
+ reader.readFloat(); // should throw
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test(expected = ParquetDecodingException.class)
+ public void testFloatSkipPastEnd() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(2, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ reader.skip(3); // should throw - only 2 values
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Hand-Crafted Binary → Reader (independent of writer) ==========
+
+ /**
+ * Construct ALP-encoded bytes manually per the spec and verify the reader
+ * decodes them correctly. This catches symmetric bugs where writer and reader
+ * both implement the same wrong thing.
+ *
+ * <p>Test vector: 3 float values {100.0f, 200.0f, 300.0f} with vectorSize=8.
+ * With exponent=0, factor=0: encoded = {100, 200, 300}.
+ * FOR min = 100, deltas = {0, 100, 200}, maxDelta = 200, bitWidth = 8.
+ * Packed: 3 values in 1 partial group of 8, padded with zeros.
+ * Packed size = ceil(3 * 8 / 8) = 3 bytes.
+ */
+ @Test
+ public void testReaderWithHandCraftedBytes() throws Exception {
+ // Manually build ALP page for {100.0f, 200.0f, 300.0f}, vectorSize=8
+ int vectorSize = 8;
+ int logVectorSize = 3;
+ int numElements = 3;
+ int numVectors = 1;
+
+ // Encoding: e=0, f=0 → multiplier=1.0
+ // encoded = {100, 200, 300}
+ // FOR min = 100
+ // deltas = {0, 100, 200}
+ // maxDelta = 200, bitWidth = 8
+ int exponent = 0;
+ int factor = 0;
+ int numExceptions = 0;
+ int frameOfReference = 100;
+ int bitWidth = 8;
+
+ // Bit-pack deltas using the same BytePacker the reader will use
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(8);
+ int[] deltas = {0, 100, 200, 0, 0, 0, 0, 0}; // padded to 8
+ byte[] packed = new byte[8]; // bitWidth bytes for one group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 8 + 7) / 8; // = 3 bytes for 3 values at 8 bits
+
+ // Build the page
+ int vectorDataSize = 4 + 5 + packedSize; // AlpInfo + ForInfo + packed
+ int offsetArraySize = numVectors * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) logVectorSize);
+ page.putInt(numElements);
+
+ // Offset array (1 vector)
+ page.putInt(offsetArraySize); // offset₀ = past offset array
+
+ // Vector 0: AlpInfo (4 bytes)
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) numExceptions);
+
+ // Vector 0: ForInfo (5 bytes)
+ page.putInt(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Vector 0: Packed data (3 bytes)
+ page.put(packed, 0, packedSize);
+
+ page.flip();
+
+ // Now read it back
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Float.floatToRawIntBits(100.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(200.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(300.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ /**
+ * Hand-craft bytes with exceptions to verify reader handles them independently.
+ * Values: {1.0f, NaN, 3.0f} with vectorSize=8.
+ * e=0, f=0: encoded={1, placeholder=1, 3}, 1 exception at position 1.
+ * FOR min=1, deltas={0, 0, 2}, maxDelta=2, bitWidth=2.
+ */
+ @Test
+ public void testReaderWithHandCraftedExceptions() throws Exception {
+ int vectorSize = 8;
+ int logVectorSize = 3;
+ int numElements = 3;
+
+ int exponent = 0;
+ int factor = 0;
+ int numExceptions = 1;
+ int placeholder = 1; // first non-exception encoded value
+ int frameOfReference = 1; // min of {1, 1, 3}
+ int bitWidth = 2; // ceil(log2(2+1)) = 2
+
+ // deltas = {1-1, 1-1, 3-1} = {0, 0, 2}, padded to 8: {0,0,2,0,0,0,0,0}
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(2);
+ int[] deltas = {0, 0, 2, 0, 0, 0, 0, 0};
+ byte[] packed = new byte[2]; // bitWidth=2 bytes for a group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 2 + 7) / 8; // = 1 byte
+
+ // Vector data: AlpInfo(4) + ForInfo(5) + packed(1) + excPos(2) + excVal(4) = 16
+ int vectorDataSize = 4 + 5 + packedSize + 2 + 4;
+ int offsetArraySize = 1 * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) logVectorSize);
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize);
+
+ // AlpInfo
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) numExceptions);
+
+ // ForInfo
+ page.putInt(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Packed data
+ page.put(packed, 0, packedSize);
+
+ // Exception positions (uint16 LE)
+ page.putShort((short) 1); // position 1
+
+ // Exception values (float32 LE)
+ page.putFloat(Float.NaN);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ float val1 = reader.readFloat();
+ assertTrue("Expected NaN at position 1", Float.isNaN(val1));
+ assertEquals(Float.floatToRawIntBits(3.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ /**
+ * Hand-craft double ALP bytes and verify reader independence.
+ * Values: {10.0, 20.0, 30.0}, vectorSize=8, e=0, f=0.
+ * encoded={10,20,30}, FOR min=10, deltas={0,10,20}, maxDelta=20, bitWidth=5.
+ */
+ @Test
+ public void testDoubleReaderWithHandCraftedBytes() throws Exception {
+ int numElements = 3;
+ int exponent = 0;
+ int factor = 0;
+ long frameOfReference = 10;
+ int bitWidth = 5; // ceil(log2(20+1)) = 5
+
+ org.apache.parquet.column.values.bitpacking.BytePackerForLong packer =
+ Packer.LITTLE_ENDIAN.newBytePackerForLong(5);
+ long[] deltas = {0, 10, 20, 0, 0, 0, 0, 0};
+ byte[] packed = new byte[5]; // bitWidth bytes for group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 5 + 7) / 8; // = 2 bytes
+
+ int vectorDataSize = 4 + 9 + packedSize; // AlpInfo + DoubleForInfo + packed
+ int offsetArraySize = 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) 3); // log2(8)
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize);
+
+ // AlpInfo
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) 0); // no exceptions
+
+ // ForInfo (9 bytes for double)
+ page.putLong(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Packed data
+ page.put(packed, 0, packedSize);
+
+ page.flip();
+
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Double.doubleToRawLongBits(10.0), Double.doubleToRawLongBits(reader.readDouble()));
+ assertEquals(Double.doubleToRawLongBits(20.0), Double.doubleToRawLongBits(reader.readDouble()));
+ assertEquals(Double.doubleToRawLongBits(30.0), Double.doubleToRawLongBits(reader.readDouble()));
+ }
+
+ // ========== Writer Output → Hand-Verified Bytes ==========
+
+ /**
+ * Write known values, then verify the exact binary output against hand computation.
+ * This verifies the writer independently of the reader.
+ *
+ * <p>Input: {1.0f, 2.0f, 3.0f} with vectorSize=8.
+ * Expected: e=0, f=0, encoded={1,2,3}, FOR min=1, deltas={0,1,2},
+ * maxDelta=2, bitWidth=2. Packed size = ceil(3*2/8) = 1 byte.
+ */
+ @Test
+ public void testWriterOutputExactBytes() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 8);
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+ writer.writeFloat(3.0f);
+
+ byte[] bytes = writer.getBytes().toByteArray();
+ ByteBuffer buf = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ assertEquals(0, buf.get() & 0xFF); // compression_mode
+ assertEquals(0, buf.get() & 0xFF); // integer_encoding
+ assertEquals(3, buf.get() & 0xFF); // log2(8) = 3
+ assertEquals(3, buf.getInt()); // num_elements
+
+ // Offset array (1 vector)
+ int offset0 = buf.getInt();
+ assertEquals(4, offset0); // 1 * 4 = past offset array
+
+ // AlpInfo
+ int exp = buf.get() & 0xFF;
+ int fac = buf.get() & 0xFF;
+ int numExc = buf.getShort() & 0xFFFF;
+ assertEquals(0, numExc);
+
+ // ForInfo
+ int forRef = buf.getInt();
+ int bw = buf.get() & 0xFF;
+
+ // Verify: for e=0, f=0: multiplier=1.0
+ // encoded = {1, 2, 3}, min = 1, deltas = {0, 1, 2}, bitWidth = 2
+ assertEquals(1, forRef); // frame of reference = 1
+ assertEquals(2, bw); // bitWidth for max delta 2
+
+ // Packed data: ceil(3 * 2 / 8) = 1 byte
+ // deltas {0, 1, 2} at bitWidth=2:
+ // Use BytePacker to compute expected packed byte
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(2);
+ int[] expectedDeltas = {0, 1, 2, 0, 0, 0, 0, 0};
+ byte[] expectedPacked = new byte[2]; // bitWidth=2
+ packer.pack8Values(expectedDeltas, 0, expectedPacked, 0);
+
+ byte actualPackedByte = buf.get();
+ assertEquals("packed byte", expectedPacked[0], actualPackedByte);
+
+ // Should have consumed all bytes
+ assertEquals("all bytes consumed", bytes.length, buf.position());
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== NaN Bit Pattern Preservation ==========
+
+ /**
+ * Verify that different NaN bit patterns survive the round-trip exactly.
+ * Java allows multiple NaN representations. Our exception handling must
+ * preserve the exact bit pattern, not normalize to Float.NaN.
+ */
+ @Test
+ public void testNaNBitPatternPreservation() throws Exception {
+ // Standard NaN
+ float standardNaN = Float.NaN; // 0x7FC00000
+ // Signaling NaN (different bit pattern)
+ float signalingNaN = Float.intBitsToFloat(0x7F800001);
+ // Negative NaN
+ float negativeNaN = Float.intBitsToFloat(0xFFC00000);
+ // Another NaN variant
+ float customNaN = Float.intBitsToFloat(0x7FFFFFFF);
+
+ assertTrue(Float.isNaN(standardNaN));
+ assertTrue(Float.isNaN(signalingNaN));
+ assertTrue(Float.isNaN(negativeNaN));
+ assertTrue(Float.isNaN(customNaN));
+
+ float[] values = {1.0f, standardNaN, 2.0f, signalingNaN, 3.0f, negativeNaN, 4.0f, customNaN};
+
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float actual = reader.readFloat();
+ assertEquals(
+ "Bit pattern mismatch at index " + i + " (0x"
+ + Integer.toHexString(Float.floatToRawIntBits(values[i])) + " vs 0x"
+ + Integer.toHexString(Float.floatToRawIntBits(actual)) + ")",
+ Float.floatToRawIntBits(values[i]),
+ Float.floatToRawIntBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /**
+ * Same test for double NaN bit patterns.
+ */
+ @Test
+ public void testDoubleNaNBitPatternPreservation() throws Exception {
+ double standardNaN = Double.NaN;
+ double signalingNaN = Double.longBitsToDouble(0x7FF0000000000001L);
+ double negativeNaN = Double.longBitsToDouble(0xFFF8000000000000L);
+ double customNaN = Double.longBitsToDouble(0x7FFFFFFFFFFFFFFFL);
+
+ double[] values = {1.0, standardNaN, 2.0, signalingNaN, 3.0, negativeNaN, 4.0, customNaN};
+
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ double actual = reader.readDouble();
+ assertEquals(
+ "Bit pattern mismatch at index " + i + " (0x"
+ + Long.toHexString(Double.doubleToRawLongBits(values[i])) + " vs 0x"
+ + Long.toHexString(Double.doubleToRawLongBits(actual)) + ")",
+ Double.doubleToRawLongBits(values[i]),
+ Double.doubleToRawLongBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Negative Zero Bit-Exact Roundtrip ==========
+
+ /**
+ * Verify that -0.0f roundtrips as -0.0f (bit pattern 0x80000000),
+ * not as +0.0f (bit pattern 0x00000000).
+ */
+ @Test
+ public void testNegativeZeroBitExact() throws Exception {
+ float negZero = -0.0f;
+ float posZero = 0.0f;
+
+ // Sanity: they are == but have different bit patterns
+ assertTrue(negZero == posZero);
+ assertNotEquals(Float.floatToRawIntBits(negZero), Float.floatToRawIntBits(posZero));
+
+ float[] values = {posZero, negZero, 1.0f, negZero};
+
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float actual = reader.readFloat();
+ assertEquals(
+ "Bit pattern at index " + i,
+ Float.floatToRawIntBits(values[i]),
+ Float.floatToRawIntBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Extreme Float Values ==========
+
+ /**
+ * Test subnormal floats, Float.MIN_VALUE, Float.MAX_VALUE, Float.MIN_NORMAL.
+ * These should all either encode losslessly or be stored as exceptions.
+ */
+ @Test
+ public void testExtremeFloatValues() throws Exception {
+ float[] values = {
+ Float.MIN_VALUE, // smallest positive subnormal: 1.4e-45
+ Float.MIN_NORMAL, // smallest positive normal: 1.17549435e-38
+ Float.MAX_VALUE, // 3.4028235e38
+ -Float.MAX_VALUE, // most negative
+ 1.17549435e-38f, // near MIN_NORMAL
+ 3.4028234e38f, // near MAX_VALUE
+ 1.0e-10f, // very small positive
+ 1.0e10f, // large positive
+ };
+ roundTripFloat(values);
+ }
+
+ /**
+ * Test extreme double values.
+ */
+ @Test
+ public void testExtremeDoubleValues() throws Exception {
+ double[] values = {
+ Double.MIN_VALUE, // smallest positive subnormal
+ Double.MIN_NORMAL, // smallest positive normal
+ Double.MAX_VALUE, // largest positive
+ -Double.MAX_VALUE, // most negative
+ 1.0e-100, // very small
+ 1.0e100, // very large
+ };
+ roundTripDouble(values);
+ }
+
+ // ========== Preset Caching Under Distribution Change ==========
+
+ /**
+ * Write >8 vectors where the optimal (e,f) changes after the sampling phase.
+ * First 8 vectors: monetary data (e=2, f=0 optimal).
+ * Remaining vectors: large integers (e=0, f=0 optimal).
+ * Verify all values survive round-trip despite the distribution change.
+ */
+ @Test
+ public void testPresetCachingWithDistributionChange() throws Exception {
+ int vectorSize = 8; // small vectors to get to 8 quickly
+ int samplingVectors = 8;
+ int postSamplingVectors = 4;
+ int totalValues = vectorSize * (samplingVectors + postSamplingVectors);
+
+ float[] values = new float[totalValues];
+
+ // First 8 vectors: decimal monetary values (best with e=2, f=0)
+ for (int i = 0; i < vectorSize * samplingVectors; i++) {
+ values[i] = 10.00f + (i % 100) * 0.01f; // 10.00, 10.01, 10.02, ...
+ }
+
+ // Last 4 vectors: whole numbers (best with e=0, f=0)
+ for (int i = vectorSize * samplingVectors; i < totalValues; i++) {
+ values[i] = (float) (i * 1000);
+ }
+
+ roundTripFloat(values, vectorSize);
+ }
+
+ /**
+ * Same test for doubles.
+ */
+ @Test
+ public void testDoublePresetCachingWithDistributionChange() throws Exception {
+ int vectorSize = 8;
+ int totalValues = vectorSize * 12; // 8 sampling + 4 post
+
+ double[] values = new double[totalValues];
+
+ for (int i = 0; i < vectorSize * 8; i++) {
+ values[i] = 10.00 + (i % 100) * 0.01;
+ }
+ for (int i = vectorSize * 8; i < totalValues; i++) {
+ values[i] = (double) (i * 1000);
+ }
+
+ roundTripDouble(values, vectorSize);
+ }
+
+ // ========== Multi-Vector Hand-Crafted Binary ==========
+
+ /**
+ * Hand-craft a 2-vector page to verify offset array navigation works.
+ * 10 values with vectorSize=8: vector 0 has values 1-8, vector 1 has 9-10.
+ */
+ @Test
+ public void testMultiVectorHandCrafted() throws Exception {
+ int vectorSize = 8;
+ int numElements = 10;
+ int numVectors = 2;
+
+ // Vector 0: {1,2,3,4,5,6,7,8} → e=0,f=0, encoded={1..8}, FOR min=1, deltas={0..7}
+ // maxDelta=7, bitWidth=3. Packed: 8 values at 3 bits = 3 bytes exactly.
+ int v0BitWidth = 3;
+ BytePacker packer3 = Packer.LITTLE_ENDIAN.newBytePacker(3);
+ int[] v0Deltas = {0, 1, 2, 3, 4, 5, 6, 7};
+ byte[] v0Packed = new byte[3];
+ packer3.pack8Values(v0Deltas, 0, v0Packed, 0);
+ int v0PackedSize = 3; // (8*3+7)/8 = 3
+
+ // Vector 1: {9, 10} → e=0,f=0, encoded={9,10}, FOR min=9, deltas={0,1}
+ // maxDelta=1, bitWidth=1. Packed: ceil(2*1/8) = 1 byte.
+ int v1BitWidth = 1;
+ BytePacker packer1 = Packer.LITTLE_ENDIAN.newBytePacker(1);
+ int[] v1Deltas = {0, 1, 0, 0, 0, 0, 0, 0};
+ byte[] v1Packed = new byte[1];
+ packer1.pack8Values(v1Deltas, 0, v1Packed, 0);
+ int v1PackedSize = 1; // (2*1+7)/8 = 1
+
+ int v0DataSize = 4 + 5 + v0PackedSize; // 12
+ int v1DataSize = 4 + 5 + v1PackedSize; // 10
+ int offsetArraySize = numVectors * 4; // 8
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + v0DataSize + v1DataSize)
+ .order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) 3); // log2(8)
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize); // vector 0 starts at 8
+ page.putInt(offsetArraySize + v0DataSize); // vector 1 starts at 8+12=20
+
+ // Vector 0
+ page.put((byte) 0); // exponent
+ page.put((byte) 0); // factor
+ page.putShort((short) 0); // exceptions
+ page.putInt(1); // FOR = 1
+ page.put((byte) v0BitWidth);
+ page.put(v0Packed, 0, v0PackedSize);
+
+ // Vector 1
+ page.put((byte) 0); // exponent
+ page.put((byte) 0); // factor
+ page.putShort((short) 0); // exceptions
+ page.putInt(9); // FOR = 9
+ page.put((byte) v1BitWidth);
+ page.put(v1Packed, 0, v1PackedSize);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ for (int i = 0; i < numElements; i++) {
+ float expected = (float) (i + 1);
+ float actual = reader.readFloat();
+ assertEquals(
+ "Value mismatch at index " + i,
+ Float.floatToRawIntBits(expected),
+ Float.floatToRawIntBits(actual));
+ }
+ }
+
+ // ========== Skip Over Entire Vector Without Decoding ==========
+
+ /**
+ * Verify that skipping an entire vector and reading the next one works.
+ * Uses hand-crafted bytes to ensure independence from writer.
+ */
+ @Test
+ public void testSkipEntireVectorHandCrafted() throws Exception {
+ // Reuse the multi-vector setup: {1..8} in vector 0, {9, 10} in vector 1
+ int vectorSize = 8;
+ int numElements = 10;
+ int numVectors = 2;
+
+ BytePacker packer3 = Packer.LITTLE_ENDIAN.newBytePacker(3);
+ int[] v0Deltas = {0, 1, 2, 3, 4, 5, 6, 7};
+ byte[] v0Packed = new byte[3];
+ packer3.pack8Values(v0Deltas, 0, v0Packed, 0);
+
+ BytePacker packer1 = Packer.LITTLE_ENDIAN.newBytePacker(1);
+ int[] v1Deltas = {0, 1, 0, 0, 0, 0, 0, 0};
+ byte[] v1Packed = new byte[1];
+ packer1.pack8Values(v1Deltas, 0, v1Packed, 0);
+
+ int v0DataSize = 4 + 5 + 3;
+ int v1DataSize = 4 + 5 + 1;
+ int offsetArraySize = numVectors * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + v0DataSize + v1DataSize)
+ .order(ByteOrder.LITTLE_ENDIAN);
+
+ page.put((byte) 0).put((byte) 0).put((byte) 3);
+ page.putInt(numElements);
+ page.putInt(offsetArraySize);
+ page.putInt(offsetArraySize + v0DataSize);
+
+ // Vector 0
+ page.put((byte) 0).put((byte) 0).putShort((short) 0);
+ page.putInt(1).put((byte) 3);
+ page.put(v0Packed, 0, 3);
+
+ // Vector 1
+ page.put((byte) 0).put((byte) 0).putShort((short) 0);
+ page.putInt(9).put((byte) 1);
+ page.put(v1Packed, 0, 1);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ // Skip entire vector 0 (8 values)
+ reader.skip(8);
+
+ // Read from vector 1
+ assertEquals(Float.floatToRawIntBits(9.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(10.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ // ========== Vector Size Validation ==========
+
+ /**
+ * Verify that vectorSize=65536 is rejected because num_exceptions (uint16)
+ * cannot represent 65536, which would cause silent data corruption if all
+ * values in a vector are exceptions.
+ */
+ @Test(expected = IllegalArgumentException.class)
+ public void testVectorSize65536Rejected() throws Exception {
+ new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 65536);
+ }
+
+ /**
+ * Verify that vectorSize=32768 (max allowed) works correctly.
+ */
+ @Test
+ public void testVectorSize32768Works() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 32768);
+ // Write a small number of values (partial vector)
+ for (int i = 0; i < 10; i++) {
+ writer.writeFloat(i * 1.0f);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(10, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < 10; i++) {
+ assertEquals(
+ Float.floatToRawIntBits(i * 1.0f),
+ Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Factor > 0 Tests ==========
+
+ /**
+ * Data where the best (e,f) combo has factor > 0.
+ * Values like 1234.56 have digits on both sides of the decimal point.
+ * With e=4,f=2: 1234.56 * 10000 / 100 = 123456 — exact integer.
+ * This exercises the division path in the encode/decode formula.
+ */
+ @Test
+ public void testDoubleFactorGreaterThanZero() throws Exception {
+ // These values are designed so the best combo has f > 0
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 1000.0 + i * 0.01; // 1000.00, 1000.01, ...
+ }
+
+ // Verify the best params actually have f > 0 or f == 0 with low exceptions
+ AlpEncoderDecoder.EncodingParams params =
+ AlpEncoderDecoder.findBestDoubleParams(values, 0, values.length);
+ // Regardless of the chosen f, the roundtrip must be lossless
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatFactorGreaterThanZero() throws Exception {
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 1000.0f + i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Overflow Boundary Tests ==========
+
+ /**
+ * Values near the encoding limits that test the overflow check.
+ * The old code used Long.MAX_VALUE (which rounds to 2^63 as double),
+ * the fix uses ENCODING_UPPER_LIMIT = 9223372036854774784.0.
+ */
+ @Test
+ public void testDoubleNearOverflowBoundary() throws Exception {
+ double[] values = {
+ 9.2e18, // near Long.MAX_VALUE
+ -9.2e18, // near Long.MIN_VALUE
+ 1.7e308, // near Double.MAX_VALUE
+ -1.7e308, // near -Double.MAX_VALUE
+ 4.9e-324, // Double.MIN_VALUE (subnormal)
+ 1.0e18, // large but in range
+ -1.0e18, // large negative but in range
+ 0.0, // normal
+ 1.23, // normal
+ };
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatNearOverflowBoundary() throws Exception {
+ float[] values = {
+ 2.1e9f, // near Integer.MAX_VALUE
+ -2.1e9f, // near Integer.MIN_VALUE
+ 3.4e38f, // near Float.MAX_VALUE
+ -3.4e38f, // near -Float.MAX_VALUE
+ 1.4e-45f, // Float.MIN_VALUE (subnormal)
+ 1.0e9f, // large but in range
+ 0.0f, // normal
+ 1.23f, // normal
+ };
+ roundTripFloat(values);
+ }
+
+ // ========== Large Scale Stress Test ==========
+
+ @Test
+ public void testDoubleLargeScale100K() throws Exception {
+ Random rand = new Random(12345);
+ double[] values = new double[100_000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 10000.0 - 5000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatLargeScale100K() throws Exception {
+ Random rand = new Random(12345);
+ float[] values = new float[100_000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 10000.0f - 5000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== All Exceptions in Large Vector ==========
+
+ /**
+ * A full vector where every value is an exception (NaN/Inf/-0.0).
+ * This exercises the case where placeholder search finds no valid value.
+ */
+ @Test
+ public void testFloatEntireVectorAllExceptions() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE + 5]; // partial second vector
+ for (int i = 0; i < values.length; i++) {
+ switch (i % 4) {
+ case 0: values[i] = Float.NaN; break;
+ case 1: values[i] = Float.POSITIVE_INFINITY; break;
+ case 2: values[i] = Float.NEGATIVE_INFINITY; break;
+ case 3: values[i] = -0.0f; break;
+ }
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleEntireVectorAllExceptions() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE + 5];
+ for (int i = 0; i < values.length; i++) {
+ switch (i % 4) {
+ case 0: values[i] = Double.NaN; break;
+ case 1: values[i] = Double.POSITIVE_INFINITY; break;
+ case 2: values[i] = Double.NEGATIVE_INFINITY; break;
+ case 3: values[i] = -0.0; break;
+ }
+ }
+ roundTripDouble(values);
+ }
+
Review Comment:
May want to add a case for numElements < Value count to cover the case where some values where null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504172424
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1066 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {
+ "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
+ if (dir == null) dir = System.getenv("ALP_TEST_DATA_DIR");
+ if (dir != null && new File(dir).isDirectory()) return Paths.get(dir);
+ // Default: alp-test-data/ relative to project root (two levels up from target/test-classes)
+ java.nio.file.Path candidate =
+ Paths.get(System.getProperty("user.dir")).resolve("alp-test-data");
+ return candidate.toFile().isDirectory() ? candidate : null;
+ }
+
+ private java.nio.file.Path getSingleTestFile() {
+ String filePath = System.getProperty("ALP_TEST_FILE");
+ if (filePath == null) filePath = System.getenv("ALP_TEST_FILE");
+ if (filePath != null && new File(filePath).exists()) return Paths.get(filePath);
+ return null;
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ @Test
+ public void testReadSingleAlpFile() throws IOException {
+ java.nio.file.Path file = getSingleTestFile();
+ assumeTrue("ALP_TEST_FILE not set or file does not exist, skipping", file != null);
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected at least one row in " + file, rows.size() > 0);
+ LOG.info("testReadSingleAlpFile: read {} rows from {}", rows.size(), file.getFileName());
+ }
+
+ @Test
+ public void testReadCppDoubleFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_DOUBLE_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppDoubleFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ @Test
+ public void testReadCppFloatFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_FLOAT_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppFloatFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ /** Verify no NaN/Inf corruption in float/double columns across all test files. */
+ @Test
+ public void testNoCorruptionInCppFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ int nanCount = 0;
+ int infCount = 0;
+ int totalValues = 0;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < pages.getRowCount(); i++) {
+ Group row = recordReader.read();
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ try {
+ if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float v = row.getFloat(f, 0);
+ totalValues++;
+ if (Float.isNaN(v)) nanCount++;
+ if (Float.isInfinite(v)) infCount++;
+ } else if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double v = row.getDouble(f, 0);
+ totalValues++;
+ if (Double.isNaN(v)) nanCount++;
+ if (Double.isInfinite(v)) infCount++;
+ }
+ } catch (Exception ignored) {
+ }
+ }
+ }
+ }
+ }
+ LOG.info(
+ "{}: {} values, {} NaN, {} Inf", filename, totalValues, nanCount, infCount);
+ assertEquals("Unexpected NaN in " + filename, 0, nanCount);
+ assertEquals("Unexpected Inf in " + filename, 0, infCount);
+ }
+ }
+
+ /** Schema inspection: log encoding types used in each column. */
+ @Test
+ public void testLogSchemaAndEncodings() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ LOG.info("=== {} ===", filename);
+ LOG.info("Schema: {}", schema);
+ if (!footer.getBlocks().isEmpty()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col :
+ footer.getBlocks().get(0).getColumns()) {
+ LOG.info(
+ " column={} encodings={} compression={}",
+ col.getPath(),
+ col.getEncodings(),
+ col.getCodec());
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not read metadata for {}: {}", filename, e.getMessage());
+ }
+ }
+ // This test always passes — it's for inspection
+ assertTrue(true);
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { "
+ + "required double double_col; "
+ + "required float float_col; "
+ + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch for " + version, DOUBLE_VALUES.length, rows.size());
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i + " for " + version,
+ DOUBLE_VALUES[i],
+ rows.get(i).getDouble("double_col", 0),
+ 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i + " for " + version,
+ FLOAT_VALUES[i],
+ rows.get(i).getFloat("float_col", 0),
+ 0.0f);
+ }
+ LOG.info("writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}", version, rows.size(), outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(4096)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch at vectorSize=4096", rowCount, rows.size());
+ for (int i = 0; i < rowCount; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i, doubles[i], rows.get(i).getDouble("double_col", 0), 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i, floats[i], rows.get(i).getFloat("float_col", 0), 0.0f);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
+ * V2 page headers include the encoding value directly; ALP = 10 is supported via the build-time
+ * patch to the generated Encoding enum in parquet-format-structures.
+ */
+ @Test
+ public void testJavaWriteAlpV2Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_2_0);
+ }
+
+ /**
+ * Writes ALP-encoded V1 and V2 parquet files from Java and verifies that pyarrow (Python Arrow)
+ * can read them back correctly. Skipped if python3/pyarrow is not available in the environment.
+ *
+ * <p>This test addresses cross-language compatibility: Java writes, another implementation reads.
+ */
+ @Test
+ public void testJavaWrittenAlpFilesReadableByPyarrow() throws IOException, InterruptedException {
+ // Check python3 is available
+ Process check = new ProcessBuilder("python3", "-c", "import pyarrow.parquet")
+ .redirectErrorStream(true)
+ .start();
+ assumeTrue("python3/pyarrow not available, skipping cross-language test", check.waitFor() == 0);
+
+ for (WriterVersion version : new WriterVersion[] {WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0}) {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_xcompat_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ // Build a python snippet that reads the file and asserts row count and values
+ StringBuilder pyScript = new StringBuilder();
+ pyScript.append("import pyarrow.parquet as pq, sys\n");
+ pyScript.append("t = pq.read_table('").append(outPath.toAbsolutePath()).append("')\n");
+ pyScript.append("assert len(t) == ").append(DOUBLE_VALUES.length).append(", f'row count mismatch: {len(t)}'\n");
+ pyScript.append("dc = t.column('double_col').to_pylist()\n");
+ pyScript.append("fc = t.column('float_col').to_pylist()\n");
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ pyScript.append("assert abs(dc[").append(i).append("] - ").append(DOUBLE_VALUES[i])
+ .append(") < 1e-9, f'double mismatch at ").append(i).append(": {dc[").append(i).append("]}'\n");
+ pyScript.append("assert abs(fc[").append(i).append("] - ").append(FLOAT_VALUES[i])
+ .append(") < 1e-4, f'float mismatch at ").append(i).append(": {fc[").append(i).append("]}'\n");
+ }
+ pyScript.append("print('OK: " + version.name() + " " + DOUBLE_VALUES.length + " rows verified by pyarrow')\n");
+
+ Process proc = new ProcessBuilder("python3", "-c", pyScript.toString())
+ .redirectErrorStream(true)
+ .start();
+ String output = new String(proc.getInputStream().readAllBytes());
+ int exitCode = proc.waitFor();
+ LOG.info("pyarrow cross-compat [{}]: {}", version, output.trim());
+ // pyarrow may not yet support reading ALP (Arrow C++ PR #48345 in progress).
+ // Skip rather than fail so the test becomes a passing signal once pyarrow adds ALP support.
+ assumeTrue(
+ "pyarrow does not yet support reading ALP encoding (Arrow C++ PR #48345 pending): "
+ + output,
+ !output.contains("Unknown encoding type"));
+ assertEquals(
+ "pyarrow failed to read Java-written ALP file (" + version + "): " + output, 0, exitCode);
+ }
+ }
+
+ // TODO: Uncomment once parquet-testing PR #100 merges
+ /*
+ @Test
+ public void testReadAlpFromParquetTesting() throws IOException {
+ // InterOpTester will auto-download from parquet-testing
+ }
+ */
+
+ // ---------------------------------------------------------------------------
+ // Fixture generator: reads the C++-written ALP files from parquet-testing
+ // PR #100 and re-encodes each as Java ALP at two vector sizes (1024 and 4096).
+ // Outputs land in ALP_OUTPUT_DIR (default: ${user.dir}/alp-java-generated/)
+ // and are intended for submission back to parquet-testing PR #100 so the
+ // C++/Rust/Go readers can verify cross-language compatibility against Java
+ // writes at both vector sizes.
+ // ---------------------------------------------------------------------------
+
+ private static final String[] SOURCE_FILES = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ private static final int[] GENERATOR_VECTOR_SIZES = {1024, 4096};
+ private static final WriterVersion[] GENERATOR_PAGE_VERSIONS = {
+ WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0
+ };
+
+ /**
+ * Resolves the explicit output directory from {@code ALP_OUTPUT_DIR} (system property or env var)
+ * and creates it if missing. Returns {@code null} when the variable is unset — callers should
+ * {@code assumeTrue} on the result so the test skips cleanly rather than writing stray files
+ * into the module tree (which would fail the RAT license check on subsequent {@code mvn test}
+ * runs in the same module).
+ */
+ private java.nio.file.Path getOutputDir() throws IOException {
+ String dir = System.getProperty("ALP_OUTPUT_DIR");
+ if (dir == null) dir = System.getenv("ALP_OUTPUT_DIR");
+ if (dir == null) return null;
+ java.nio.file.Path target = Paths.get(dir);
+ java.nio.file.Files.createDirectories(target);
+ return target;
+ }
+
+ /**
+ * Read all rows of the given source file and return them as a list of value-only Group rows
+ * keyed by the source's schema. Used for both the copy and the verification round-trip.
+ */
+ private List<Group> readGroupsWithSchema(java.nio.file.Path source, MessageType[] outSchema) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(source))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ outSchema[0] = schema;
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ /**
+ * Generates Java-written ALP fixtures from the four PR #100 source datasets across two axes:
+ * page version (V1, V2) and vectorSize (1024, 4096). Each output is verified bit-exact against
+ * its source. Produces 16 files total:
+ * {@code alp_java_{spotify1,arade,float_spotify1,float_arade}_v{1,2}_vs{1024,4096}.parquet}.
+ *
+ * <p>Page version is orthogonal to ALP encoding (the page version difference lives in the
+ * parquet protocol layer, not in the ALP payload), but covering both axes makes the fixture
+ * set fully symmetric for cross-language compatibility — every reader can verify it handles
+ * Java-written ALP regardless of how the surrounding pages are framed.
+ *
+ * <p>To run: clone https://github.com/prtkgaur/parquet-testing/tree/alpFloatingPointDataset
+ * and point ALP_TEST_DATA_DIR at its {@code data/} directory. Set ALP_OUTPUT_DIR to choose
+ * where to write the generated fixtures; if unset, this test skips.
+ */
+ @Test
+ public void generateAlpFixturesAtMultipleVectorSizes() throws IOException {
+ java.nio.file.Path sourceDir = getTestDataDir();
+ assumeTrue("ALP source dir not found, skipping fixture generator", sourceDir != null);
+
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping fixture generator", outDir != null);
+ LOG.info("Generating ALP fixtures to {}", outDir);
+
+ int expectedFiles =
+ SOURCE_FILES.length * GENERATOR_PAGE_VERSIONS.length * GENERATOR_VECTOR_SIZES.length;
+ int generated = 0;
+ for (String sourceFile : SOURCE_FILES) {
+ java.nio.file.Path source = sourceDir.resolve(sourceFile);
+ assumeTrue("Source missing: " + source, source.toFile().exists());
+
+ MessageType[] schemaHolder = new MessageType[1];
+ List<Group> sourceRows = readGroupsWithSchema(source, schemaHolder);
+ MessageType schema = schemaHolder[0];
+
+ String stem = sourceFile.replace("alp_", "").replace(".parquet", "");
+
+ for (WriterVersion pageVersion : GENERATOR_PAGE_VERSIONS) {
+ String pageTag = (pageVersion == WriterVersion.PARQUET_1_0) ? "v1" : "v2";
+ for (int vectorSize : GENERATOR_VECTOR_SIZES) {
+ java.nio.file.Path outPath =
+ outDir.resolve(
+ "alp_java_" + stem + "_" + pageTag + "_vs" + vectorSize + ".parquet");
+ java.nio.file.Files.deleteIfExists(outPath);
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(pageVersion)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(vectorSize)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (Group row : sourceRows) {
+ writer.write(row);
+ }
+ }
+
+ // Verify: read back and compare against the source row-by-row
+ List<Group> roundTrip = readGroupsWithSchema(outPath, new MessageType[1]);
+ assertEquals(
+ "Row count mismatch for " + outPath.getFileName(),
+ sourceRows.size(),
+ roundTrip.size());
+ int fieldCount = schema.getFieldCount();
+ for (int i = 0; i < sourceRows.size(); i++) {
+ for (int f = 0; f < fieldCount; f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ String fieldName = schema.getType(f).getName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double expected = sourceRows.get(i).getDouble(f, 0);
+ double actual = roundTrip.get(i).getDouble(f, 0);
+ long eb = Double.doubleToRawLongBits(expected);
+ long ab = Double.doubleToRawLongBits(actual);
+ assertEquals(
+ "double bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float expected = sourceRows.get(i).getFloat(f, 0);
+ float actual = roundTrip.get(i).getFloat(f, 0);
+ int eb = Float.floatToRawIntBits(expected);
+ int ab = Float.floatToRawIntBits(actual);
+ assertEquals(
+ "float bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ }
+ }
+ }
+ long bytes = outPath.toFile().length();
+ LOG.info(
+ " wrote {} ({} rows, {} bytes, pageVersion={}, vectorSize={})",
+ outPath.getFileName(),
+ sourceRows.size(),
+ bytes,
+ pageTag,
+ vectorSize);
+ generated++;
+ }
+ }
+ }
+ assertEquals(
+ "Expected to generate " + expectedFiles + " fixture files", expectedFiles, generated);
+ LOG.info("Generated {} ALP fixture files to {}", generated, outDir);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Reader-only validation: opens each Java-written fixture and asserts the
+ // reader path works end-to-end (metadata declares ALP, every row decodes
+ // without error). Separate from the generator's own round-trip check so it
+ // surfaces as a distinct signal — useful when the fixtures already exist on
+ // disk and you want to validate just the read path.
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void readAllFixtureFilesIndependently() throws IOException {
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping reader-only test", outDir != null);
+ File[] files =
+ outDir.toFile().listFiles((f, n) -> n.startsWith("alp_java_") && n.endsWith(".parquet"));
+ assumeTrue("No fixture files in " + outDir, files != null && files.length > 0);
+ java.util.Arrays.sort(files, (a, b) -> a.getName().compareTo(b.getName()));
+
+ int filesRead = 0;
+ int totalChunks = 0;
+ int alpChunks = 0;
+ long totalRows = 0;
+ for (File pf : files) {
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(pf.toPath()))) {
+ ParquetMetadata footer = reader.getFooter();
+ // Layer 1: every column chunk declares ALP
+ for (org.apache.parquet.hadoop.metadata.BlockMetaData block : footer.getBlocks()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col : block.getColumns()) {
+ totalChunks++;
+ if (col.getEncodings().contains(Encoding.ALP)) alpChunks++;
+ }
+ }
+ // Layer 2: every row actually decodes (no exceptions thrown, count matches footer)
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ long rowsDecoded = 0;
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ Group row = recordReader.read();
+ // Touch every present field so any decode-time exception surfaces.
+ // Optional/null fields have repetitionCount=0 — skip those rather than throw.
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ if (row.getFieldRepetitionCount(f) == 0) continue;
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ row.getDouble(f, 0);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ row.getFloat(f, 0);
+ }
+ }
+ rowsDecoded++;
+ }
+ }
+ totalRows += rowsDecoded;
+ filesRead++;
+ }
+ }
+ assertEquals("Every column chunk should declare ALP", totalChunks, alpChunks);
+ LOG.info(
+ "readAllFixtureFilesIndependently: {} files, {} chunks ({} ALP), {} rows decoded",
+ filesRead, totalChunks, alpChunks, totalRows);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Corner-case fixture per parquet-testing issue #105. Writes a single small
+ // file with one ALP-encoded column per corner case (no exceptions, 1 exception,
+ // all exceptions, NaN/Inf/-0.0, nulls, constant/bit_width=0, multi-vector with
+ // differing exponents) for both f32 and f64, then reads back and verifies
+ // every value bit-exactly against the expected pattern.
+ //
+ // The output file (alp_java_cornercases.parquet) is intended as a candidate
+ // fixture for parquet-testing PR #100 once Prateek confirms naming and design.
+ // ---------------------------------------------------------------------------
+
+ private static final int CORNER_ROWS_PER_VECTOR = 1024;
+ private static final int CORNER_VECTORS = 2;
+ private static final int CORNER_TOTAL_ROWS = CORNER_ROWS_PER_VECTOR * CORNER_VECTORS;
+
+ /** Expected values per corner-case column. Doubles use NaN to mark a null cell for sentinel use. */
+ private static class CornerCaseData {
+ double[] doubleVals;
+ float[] floatVals;
+ boolean[] isNull; // only meaningful for optional columns
+ boolean isFloat;
+ boolean isOptional;
+ String name;
+
+ static CornerCaseData doubles(String name, double[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.doubleVals = vals;
+ d.isFloat = false;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData floats(String name, float[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.floatVals = vals;
+ d.isFloat = true;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData optionalDoubles(String name, double[] vals, boolean[] nulls) {
+ CornerCaseData d = doubles(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+
+ static CornerCaseData optionalFloats(String name, float[] vals, boolean[] nulls) {
+ CornerCaseData d = floats(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+ }
+
+ private List<CornerCaseData> buildCornerCaseColumns() {
+ List<CornerCaseData> cols = new ArrayList<>();
+ int N = CORNER_TOTAL_ROWS;
+
+ // 1. f64_no_exceptions: clean 2-decimal values, both vectors low bit-width
+ double[] noExc = new double[N];
+ for (int i = 0; i < N; i++) noExc[i] = (i % 1000) / 100.0;
+ cols.add(CornerCaseData.doubles("f64_no_exceptions", noExc));
+
+ // 2. f64_one_exception_per_vector: clean values with 1 NaN per 1024-row vector
+ double[] oneExc = new double[N];
+ for (int i = 0; i < N; i++) oneExc[i] = (i % 1000) / 100.0;
+ oneExc[100] = Double.NaN;
+ oneExc[1100] = Double.NaN;
+ cols.add(CornerCaseData.doubles("f64_one_exception_per_vector", oneExc));
+
+ // 3. f64_all_nan: every value NaN — all-exception vector
+ double[] allNan = new double[N];
+ java.util.Arrays.fill(allNan, Double.NaN);
+ cols.add(CornerCaseData.doubles("f64_all_nan", allNan));
+
+ // 4. f64_nan_inf_negzero: mix of special values across two vectors
+ double[] mixed = new double[N];
+ for (int i = 0; i < N; i++) mixed[i] = (i % 100) / 10.0;
+ mixed[0] = Double.NaN;
+ mixed[1] = Double.POSITIVE_INFINITY;
+ mixed[2] = Double.NEGATIVE_INFINITY;
+ mixed[3] = -0.0;
+ mixed[1024] = Double.NaN;
+ mixed[1025] = -0.0;
+ cols.add(CornerCaseData.doubles("f64_nan_inf_negzero", mixed));
+
+ // 5. f64_constant: every value = 42.0 → max_delta=0 → bit_width=0
+ double[] constant = new double[N];
+ java.util.Arrays.fill(constant, 42.0);
+ cols.add(CornerCaseData.doubles("f64_constant", constant));
+
+ // 6. f64_different_e_f_per_vector: vector 0 uses e=2,f=0 (cents); vector 1 uses e=3,f=0 (mils)
+ double[] diff = new double[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diff[i] = (i % 100) / 100.0;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diff[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0;
+ cols.add(CornerCaseData.doubles("f64_different_ef_per_vector", diff));
+
+ // 7. f64_with_nulls: optional column, ~30% nulls scattered
+ double[] withNulls = new double[N];
+ boolean[] nullMask = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMask[i] = true;
+ } else {
+ withNulls[i] = (i % 100) / 100.0;
+ }
+ }
+ cols.add(CornerCaseData.optionalDoubles("f64_with_nulls", withNulls, nullMask));
+
+ // 8–14: f32 analogues
+ float[] noExcF = new float[N];
+ for (int i = 0; i < N; i++) noExcF[i] = (i % 1000) / 100.0f;
+ cols.add(CornerCaseData.floats("f32_no_exceptions", noExcF));
+
+ float[] oneExcF = new float[N];
+ for (int i = 0; i < N; i++) oneExcF[i] = (i % 1000) / 100.0f;
+ oneExcF[100] = Float.NaN;
+ oneExcF[1100] = Float.NaN;
+ cols.add(CornerCaseData.floats("f32_one_exception_per_vector", oneExcF));
+
+ float[] allNanF = new float[N];
+ java.util.Arrays.fill(allNanF, Float.NaN);
+ cols.add(CornerCaseData.floats("f32_all_nan", allNanF));
+
+ float[] mixedF = new float[N];
+ for (int i = 0; i < N; i++) mixedF[i] = (i % 100) / 10.0f;
+ mixedF[0] = Float.NaN;
+ mixedF[1] = Float.POSITIVE_INFINITY;
+ mixedF[2] = Float.NEGATIVE_INFINITY;
+ mixedF[3] = -0.0f;
+ mixedF[1024] = Float.NaN;
+ mixedF[1025] = -0.0f;
+ cols.add(CornerCaseData.floats("f32_nan_inf_negzero", mixedF));
+
+ float[] constantF = new float[N];
+ java.util.Arrays.fill(constantF, 42.0f);
+ cols.add(CornerCaseData.floats("f32_constant", constantF));
+
+ float[] diffF = new float[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diffF[i] = (i % 100) / 100.0f;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diffF[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0f;
+ cols.add(CornerCaseData.floats("f32_different_ef_per_vector", diffF));
+
+ float[] withNullsF = new float[N];
+ boolean[] nullMaskF = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMaskF[i] = true;
+ } else {
+ withNullsF[i] = (i % 100) / 100.0f;
+ }
+ }
+ cols.add(CornerCaseData.optionalFloats("f32_with_nulls", withNullsF, nullMaskF));
+
+ return cols;
+ }
+
+ /**
+ * Dumps the corner-case construction recipe to a CSV truth file alongside the parquet.
+ * The CSV is built directly from the expected values (not by reading the parquet back),
+ * which makes it suitable as independent ground truth for cross-language verifiers.
+ *
+ * <p>Conventions:
+ * <ul>
+ * <li>Header row matches column names.
+ * <li>Empty field marks a null cell in an optional column.
+ * <li>Special values use Java's standard toString output: {@code NaN}, {@code Infinity},
+ * {@code -Infinity}. These parse correctly via C++ {@code std::stod} / {@code std::stof}
+ * (per the standard, both are case-insensitive and accept the full word "Infinity").
+ * <li>{@code -0.0} round-trips through both Java and C++ string conversion bit-exact.
+ * </ul>
+ */
+ private void writeCornerCaseCsvTruth(java.nio.file.Path csvPath, List<CornerCaseData> cols) throws IOException {
+ try (java.io.BufferedWriter w = java.nio.file.Files.newBufferedWriter(csvPath)) {
+ // Header
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ w.write(cols.get(i).name);
+ }
+ w.write("\n");
+ // Data
+ for (int r = 0; r < CORNER_TOTAL_ROWS; r++) {
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ CornerCaseData c = cols.get(i);
+ if (c.isOptional && c.isNull[r]) {
+ // empty field = null
+ } else if (c.isFloat) {
+ w.write(Float.toString(c.floatVals[r]));
+ } else {
+ w.write(Double.toString(c.doubleVals[r]));
+ }
+ }
+ w.write("\n");
+ }
+ }
+ }
+
+ @Test
+ public void generateAndVerifyCornerCaseFixture() throws IOException {
Review Comment:
another corner case is when MIN and MAX INT/LONG values end up getting encoding in FOR, commented on the C++ test but we should ensure we have tests that cover that as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3997787303
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +624,77 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Set the full ALP configuration for FLOAT and DOUBLE columns.
+ *
+ * @param config the ALP configuration
+ * @return this builder for method chaining.
+ */
+ public Builder withAlp(AlpConfig config) {
Review Comment:
Yes, I think so, and it is in a8c584f64 as `withoutAlp(columnPath)`. The reason it was missing is that ColumnProperty treated a null per-column value as unset and fell back to the default, so it now checks containsKey instead and an explicit null overrides; every other per-column property either cannot hold a null or already defaults to one, so nothing else changes behaviour.
Two things had to come with it. The schema check was treating "column named in the ALP property" as "ALP enabled here", so disabling ALP on an int column failed with a message claiming ALP was enabled on it, and toString rendered a disabled column as null again, which undoes your other nit. Both are fixed and covered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3973409372
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1406 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.values.alp.AlpConfig;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {"alp_float_spotify1.parquet", "alp_float_arade.parquet"};
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
+ if (dir == null) dir = System.getenv("ALP_TEST_DATA_DIR");
+ if (dir != null && new File(dir).isDirectory()) return Paths.get(dir);
+ // Default: alp-test-data/ relative to project root (two levels up from target/test-classes)
+ java.nio.file.Path candidate = Paths.get(System.getProperty("user.dir")).resolve("alp-test-data");
+ return candidate.toFile().isDirectory() ? candidate : null;
+ }
+
+ private java.nio.file.Path getSingleTestFile() {
+ String filePath = System.getProperty("ALP_TEST_FILE");
+ if (filePath == null) filePath = System.getenv("ALP_TEST_FILE");
+ if (filePath != null && new File(filePath).exists()) return Paths.get(filePath);
+ return null;
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ @Test
+ public void testReadSingleAlpFile() throws IOException {
+ java.nio.file.Path file = getSingleTestFile();
+ assumeTrue(file != null, "ALP_TEST_FILE not set or file does not exist, skipping");
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected at least one row in " + file).isTrue();
+ LOG.info("testReadSingleAlpFile: read {} rows from {}", rows.size(), file.getFileName());
+ }
+
+ @Test
+ public void testReadCppDoubleFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ for (String filename : CPP_DOUBLE_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue(file.toFile().exists(), "File not found: " + file);
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected rows in " + filename).isTrue();
+ LOG.info("testReadCppDoubleFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ @Test
+ public void testReadCppFloatFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ for (String filename : CPP_FLOAT_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue(file.toFile().exists(), "File not found: " + file);
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected rows in " + filename).isTrue();
+ LOG.info("testReadCppFloatFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ /** Verify no NaN/Inf corruption in float/double columns across all test files. */
+ @Test
+ public void testNoCorruptionInCppFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ String[] allFiles = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ int nanCount = 0;
+ int infCount = 0;
+ int totalValues = 0;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < pages.getRowCount(); i++) {
+ Group row = recordReader.read();
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ try {
+ if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float v = row.getFloat(f, 0);
+ totalValues++;
+ if (Float.isNaN(v)) nanCount++;
+ if (Float.isInfinite(v)) infCount++;
+ } else if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double v = row.getDouble(f, 0);
+ totalValues++;
+ if (Double.isNaN(v)) nanCount++;
+ if (Double.isInfinite(v)) infCount++;
+ }
+ } catch (Exception ignored) {
+ }
+ }
+ }
+ }
+ }
+ LOG.info("{}: {} values, {} NaN, {} Inf", filename, totalValues, nanCount, infCount);
+ assertThat(nanCount).as("Unexpected NaN in " + filename).isEqualTo(0);
+ assertThat(infCount).as("Unexpected Inf in " + filename).isEqualTo(0);
+ }
+ }
+
+ /** Schema inspection: log encoding types used in each column. */
+ @Test
+ public void testLogSchemaAndEncodings() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ String[] allFiles = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ LOG.info("=== {} ===", filename);
+ LOG.info("Schema: {}", schema);
+ if (!footer.getBlocks().isEmpty()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col :
+ footer.getBlocks().get(0).getColumns()) {
+ LOG.info(
+ " column={} encodings={} compression={}",
+ col.getPath(),
+ col.getEncodings(),
+ col.getCodec());
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not read metadata for {}: {}", filename, e.getMessage());
+ }
+ }
+ // This test always passes — it's for inspection
+ assertThat(true).isTrue();
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { " + "required double double_col; " + "required float float_col; " + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlp()
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch for " + version).isEqualTo(DOUBLE_VALUES.length);
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i + " for " + version)
+ .isEqualTo(DOUBLE_VALUES[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i + " for " + version)
+ .isEqualTo(FLOAT_VALUES[i]);
+ }
+ LOG.info(
+ "writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}",
+ version,
+ rows.size(),
+ outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath = newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlp(new AlpConfig(4096))
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch at vectorSize=4096").isEqualTo(rowCount);
+ for (int i = 0; i < rowCount; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i)
+ .isEqualTo(doubles[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i)
+ .isEqualTo(floats[i]);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
Review Comment:
Java doc here is not in sync. One of the reasons I'm always pushing on not including details in javadocs because they changed very fast :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3579956345
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
Review Comment:
I would just add the method to BytesUtil unless you think there is something special about this class's usage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3567845033
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForFloat() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new float[capacity];
+ this.deltasBuffer = new int[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Integer.SIZE]; // max bit width for int = 32 bytes
+ }
+
Review Comment:
Applied the same changes to the float reader: bounds check, vectorNumber naming, typed getters, and the BytesUtils helpers.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > DOUBLE_MAX_EXPONENT) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double exponent " + exponent + " in vector " + vectorIdx + ", max is " + DOUBLE_MAX_EXPONENT);
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double factor " + factor + " > exponent " + exponent + " in vector " + vectorIdx);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException(
+ "Invalid ALP numExceptions " + numExceptions + " > vectorLen " + vectorLen + " in vector " + vectorIdx);
+ }
+
+ long frameOfReference = getLongLE(vectorsData, pos);
+ int bitWidth = vectorsData.get(pos + 8) & 0xFF;
+ pos += DOUBLE_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
Review Comment:
Added the bitWidth > 64 bounds check in the double reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099429898
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
Review Comment:
This could also check that the vector isn't too long because it is bounded by the length of storing every value in its original bit width.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4305080764
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_FOR_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.PACK_GROUP_SIZE;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private final long[] unpackPadBuf = new long[PACK_GROUP_SIZE];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ protected int maxExponent() {
+ return DOUBLE_MAX_EXPONENT;
+ }
+
+ @Override
+ protected String typeName() {
+ return "double";
+ }
+
+ @Override
+ protected int forInfoSize() {
+ return DOUBLE_FOR_INFO_SIZE;
+ }
+
+ @Override
+ protected int exceptionValueSize() {
+ return Double.BYTES;
+ }
+
+ @Override
+ public double readDouble() {
+ if (pageValueIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int vectorSlot = pageValueIndex % vectorSize;
Review Comment:
Fixed, in both the double and float readers.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_FOR_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.PACK_GROUP_SIZE;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private final long[] unpackPadBuf = new long[PACK_GROUP_SIZE];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ protected int maxExponent() {
+ return DOUBLE_MAX_EXPONENT;
+ }
+
+ @Override
+ protected String typeName() {
+ return "double";
+ }
+
+ @Override
+ protected int forInfoSize() {
+ return DOUBLE_FOR_INFO_SIZE;
+ }
+
+ @Override
+ protected int exceptionValueSize() {
+ return Double.BYTES;
+ }
+
+ @Override
+ public double readDouble() {
+ if (pageValueIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int vectorSlot = pageValueIndex % vectorSize;
+ pageValueIndex++;
+ return decodedValues[vectorSlot];
+ }
+
+ @Override
+ protected int decodeBody(int pos, int vectorNumber, int vectorLen, int exponent, int factor) {
+ long frameOfReference = vectorsData.getLong(pos);
+ int bitWidth = vectorsData.get(pos + Long.BYTES) & 0xFF;
+ if (bitWidth > Long.SIZE) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double bitWidth " + bitWidth + " > 64 in vector " + vectorNumber);
+ }
+ pos += DOUBLE_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
+ pos = unpackLongsWithBytePacker(vectorsData, pos, deltasBuffer, vectorLen, bitWidth);
+ } else {
+ Arrays.fill(deltasBuffer, 0, vectorLen, 0L);
+ }
+
+ for (int i = 0; i < vectorLen; i++) {
+ long encoded = deltasBuffer[i] + frameOfReference;
+ decodedValues[i] = AlpCodec.decodeDouble(encoded, exponent, factor);
+ }
+ return pos;
+ }
+
+ @Override
+ protected void applyExceptionValues(int pos, int numExceptions) {
Review Comment:
Moved reading the exception positions into this method, and the logic itself now lives in static methods (`readExceptionPositions` and `applyExceptionValues`) that take everything as arguments. The override just passes the fields in, since it can't be static itself. Same change in the float reader.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
Review Comment:
I added the first-offset check, and it now runs before the offsets array is allocated.
I kept the byte-count bound too, though. I think the math here is off: with 1024-value vectors, `num_elements = 2^31-1` is 2,097,152 vectors, and each one needs at least 17 bytes (4 for its offset plus 13 for the AlpInfo and ForInfo headers for doubles). So it's a multiplication, and a page would need about 36 MB to get past the check. Anything a 1 MB page can't actually hold gets rejected.
The two checks also catch different problems. The offset check catches garbage in the offset array, and the bound catches a truncated or forged page that claims a huge count, before we allocate based on that count. So I'd like to keep both, but let me know if you feel strongly.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
Review Comment:
Done, the loop now runs from 1 to `numVectors`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
Review Comment:
Added. One difference from the bound you suggested on `:113`: it has to leave room for exceptions, which cost 2 bytes for the position plus the value itself. Random doubles are almost all exceptions, so a 1024-value vector of them takes about 10,253 bytes even at bit width 0, and `17 + 8 * vectorSize` only allows 8,209. Using that bound breaks 20 of the existing round-trip tests. So the limit assumes every value is packed at full width and also stored as an exception. It's loose, but it won't reject a valid page, and with the extra end entry from `:176` it covers the last vector too.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
Review Comment:
Changed `vectorOffsets` to have `numVectors + 1` entries, with the last one marking the end of the vector data. Out-of-range indexes now throw `IllegalArgumentException`, so `Integer.MAX_VALUE` no longer returns `limit()`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
Review Comment:
Good point. The data size is now recorded once when the data is sliced, and nothing reads `limit()` or `remaining()` after that. We hit a similar bug earlier with absolute reads on the page header, so there's also a test now with a page buffer that doesn't start at position 0.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
Review Comment:
It's static now: `validateVectorOffsets(int[] vectorOffsets, int minVectorBytes, int maxVectorBytes)`. I didn't add a separate data size argument because the data size is now the last entry of `vectorOffsets` (from `:176`), so the last vector gets checked by the same loop. I can pass it separately if you'd rather.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-6243577437
@rdblue thanks for the review, and sorry it took me a while to get back to you. I pushed two commits: 2a9a23f8e restructures the reader to follow the spec's sections (plus the renames and static helpers), and 647198fd4 tightens the offset validation. I'm working through the rest of your comments next.
I replied on the threads those two commits cover. In one spot I went a different way: I kept the byte-count bound on `:113`, and I explained why there.
@wgtmac I opened #3862 and added it to the title.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504098383
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
Review Comment:
curious if this shows up in profiles, and if we can use the fact that vectorSize is a power of 2 to optimize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3567857469
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
Review Comment:
Correction to my previous reply: the mask is actually still here and is needed. We did switch to vectorsData.getShort(pos + 2), and getShort returns a signed short, so the & 0xFFFF reads num_exceptions as an unsigned 16-bit value. That's exactly the case you flagged, so it's correct as written rather than removed. Sorry for the mix-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3790415963
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
Review Comment:
Done — moved it to `BytesUtils` as `getWidthFromMaxLong`, the long counterpart to `getWidthFromMaxInt`, and dropped the ALP-local `bitWidthForLong`. Nothing special about the usage here: `DeltaBinaryPackingValuesWriterForLong` already hand-rolls the same `64 - Long.numberOfLeadingZeros(x)` expression, so it's generally useful rather than ALP-specific.
Two things that came out of the move:
- The old `maxDelta == 0` guard was dead code. `Long.numberOfLeadingZeros(0)` is 64, so the subtraction already returns 0. The two size estimators that inlined the same expression now call the helper directly instead of guarding with a ternary.
- The unit test moved to `TestBytesUtil` next to the int version, and picked up two cases the ALP-local test didn't cover: a bound above the int range (the reason the long version needs to exist) and a negative bound, which occupies the full 64 bits.
I left `DeltaBinaryPackingValuesWriterForLong` untouched to keep this PR scoped to ALP, but happy to switch it over here or in a follow-up, whichever you prefer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099515386
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
Review Comment:
This leaves out the most important thing, which is that this is relative to the start of the vector data section and is not an offset as defined by the spec.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099288647
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
Review Comment:
Nit: I prefer constants for values like this, like `BEFORE_FIRST_VECTOR`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-5653117709
Opened #3791 as the thrift precursor for this PR.
It bumps the inlined `parquet.thrift` from parquet-format 2.13.0 to 2.14.0 using `dev/update-parquet-thrift.sh`, which is a released tag containing ALP, so the entry comes from upstream rather than being hand written here.
One thing worth knowing, since it changes the shape of what was suggested. The bump cannot be thrift only: on a clean master it fails `testEnumEquivalence` with `No enum constant org.apache.parquet.column.Encoding.ALP`, because the generated format enum gains ALP(10) while the hand written `column.Encoding` has none, and adding that constant pulls in the ALP readers with it. So #3791 carries a temporary skip for ALP, which this PR removes.
Once #3791 merges I will rebase this one and drop the local `ALP = 10` block, the "LOCAL ADDITION" notice, and that skip. The bump also brings the FILE logical type and INT96 chronological ordering, neither of which needs Java side handling beyond one guard so a FILE annotated column keeps degrading to its physical type instead of throwing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3962574774
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +624,77 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Set the full ALP configuration for FLOAT and DOUBLE columns.
+ *
+ * @param config the ALP configuration
+ * @return this builder for method chaining.
+ */
+ public Builder withAlp(AlpConfig config) {
Review Comment:
These APIs seem a bit over-specified for what we allow?
I feel like we have
withAlp(config) // sets ALP on everywhere
withAlp(columnPath, config) // set config on one column
withAlp() // Sets ALP on with default config
withAlp(columnPath) //sets alp as on for a single column
I'm not sure we need the ability to flip the encoding back off
Then I would make sure that none of this includes implementation details (like config parameters) in the method names.
I would also consider whether we should fail if "ALP" is enabled on a non float/double column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3964216004
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -51,6 +53,8 @@ public class ParquetProperties {
public static final int DEFAULT_DICTIONARY_PAGE_SIZE = DEFAULT_PAGE_SIZE;
public static final boolean DEFAULT_IS_DICTIONARY_ENABLED = true;
public static final boolean DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED = false;
+ public static final boolean DEFAULT_IS_ALP_ENABLED = false;
+ public static final int DEFAULT_ALP_VECTOR_SIZE = AlpConfig.DEFAULT_VECTOR_SIZE;
Review Comment:
Agreed, and thanks for flagging it — there was no reason to alias the constant. Removed in 7377588ef, so the call site now reads `AlpConfig.DEFAULT_VECTOR_SIZE` directly.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConfig.java:
##########
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import java.util.Objects;
+
+/**
+ * Immutable per-column ALP encoding configuration: whether ALP is enabled and the vector size
+ * (number of values per encoded vector) to use. Bundled together so a column carries a single
+ * cohesive ALP setting rather than several independent properties.
+ */
+public final class AlpConfig {
+
+ /** Default values per encoded vector. */
+ public static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ /** ALP disabled, with the default vector size. */
+ public static final AlpConfig DISABLED = new AlpConfig(false, DEFAULT_VECTOR_SIZE);
+
+ private final boolean enabled;
+ private final int vectorSize;
+
+ /**
+ * @param enabled whether ALP encoding is enabled
+ * @param vectorSize values per encoded vector; must be a power of 2 in the supported range
+ * @throws IllegalArgumentException if {@code vectorSize} is not a supported vector size
+ */
+ public AlpConfig(boolean enabled, int vectorSize) {
Review Comment:
That separation makes sense, thank you. Fixed in e3b846a2b, where `AlpConfig` holds only the vector size and a column is ALP encoded when it has a config at all.
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -468,6 +504,8 @@ private Builder() {
DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED
? ByteStreamSplitMode.FLOATING_POINT
: ByteStreamSplitMode.NONE);
+ alp = ColumnProperty.<AlpConfig>builder()
Review Comment:
Good catch, it was silently dropping BYTE_STREAM_SPLIT. Both that conflict and ALP on a non float or double column are now rejected in `newColumnWriteStore` in e660c6cec, rather than resolved by precedence.
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +624,77 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Set the full ALP configuration for FLOAT and DOUBLE columns.
+ *
+ * @param config the ALP configuration
+ * @return this builder for method chaining.
+ */
+ public Builder withAlp(AlpConfig config) {
Review Comment:
Your sketch is what the API looks like now, thanks for laying it out: e3b846a2b collapses the six methods into `withAlp()`, `withAlp(config)`, `withAlp(columnPath)`, `withAlp(columnPath, config)` and `withoutAlp()`, with the vector size only reachable through `AlpConfig`. The non float or double check is in e660c6cec, and the one tradeoff worth flagging is that per-column disabling goes away with the boolean setters, so let me know if you would like that kept.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4305133962
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
Review Comment:
Removed it, it's computed from the vector count now. I did the same for the vector data size, which only needs to be read once when the data is sliced.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
Review Comment:
Done, the fields are initialized where they're declared now.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
Review Comment:
Done. It's `Math.toIntExact`, since `Integer` doesn't have one. I kept the byte-count bound above it, see `:113`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
Review Comment:
Added `BEFORE_FIRST_VECTOR`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
Review Comment:
Done. Both readers now keep their buffers between pages and only allocate new ones when a page needs bigger ones, so a page with a smaller vector size reuses them too.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+final class AlpConstants {
+
+ private AlpConstants() {
+ // Utility class
+ }
+
+ // Page header fields
+ static final int ALP_COMPRESSION_MODE = 0;
+ static final int ALP_INTEGER_ENCODING_FOR = 0;
+ static final int ALP_HEADER_SIZE = 7;
+
+ static final int DEFAULT_VECTOR_SIZE = 1024;
+ static final int DEFAULT_VECTOR_SIZE_LOG = 10;
+
+ // BytePacker packs/unpacks 8 values at a time (pack8Values/unpack8Values).
+ static final int PACK_GROUP_SIZE = 8;
+
+ // Capped at 15 (vectorSize=32768) because num_exceptions is uint16,
+ // so vectorSize must not exceed 65535 to avoid overflow when all values are exceptions.
+ static final int MAX_LOG_VECTOR_SIZE = 15;
+ static final int MIN_LOG_VECTOR_SIZE = 3;
+
+ static final int FLOAT_MAX_EXPONENT = 10;
+ static final int DOUBLE_MAX_EXPONENT = 18;
+
+ // Per-vector metadata sizes in bytes
+ static final int ALP_INFO_SIZE = 4; // exponent(1) + factor(1) + num_exceptions(2)
+ static final int FLOAT_FOR_INFO_SIZE = 5; // frame_of_reference(4) + bit_width(1)
+ static final int DOUBLE_FOR_INFO_SIZE = 9; // frame_of_reference(8) + bit_width(1)
+
+ /** Validates vector size: must be a power of 2 in [2^MIN_LOG .. 2^MAX_LOG]. */
+ static int validateVectorSize(int vectorSize) {
+ Preconditions.checkArgument(
+ vectorSize > 0 && (vectorSize & (vectorSize - 1)) == 0,
Review Comment:
Switched to `Integer.bitCount(vectorSize) == 1`. I kept the `vectorSize > 0` check because `bitCount` alone would accept `Integer.MIN_VALUE`. That check didn't have a test, so I added one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4305192375
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
+
+ // The packed body's width comes from the vector's own FOR header, so confirm it did not run
+ // past where the next vector begins; otherwise this vector would decode another one's bytes.
+ int vectorEnd = getVectorEndPosition(vectorNumber);
+ if (pos > vectorEnd) {
+ throw new ParquetDecodingException("ALP vector " + vectorNumber + " packed body ends at " + pos
+ + ", past the start of the next vector at " + vectorEnd);
+ }
+ if (numExceptions > 0) {
+ long exceptionBytes = (long) numExceptions * (Short.BYTES + exceptionValueSize());
+ if (pos + exceptionBytes > vectorEnd) {
+ throw new ParquetDecodingException(
+ "ALP vector " + vectorNumber + " declares " + numExceptions + " exceptions needing "
+ + exceptionBytes + " bytes, past the end of the vector at " + vectorEnd);
+ }
+ pos = readExceptionPositions(pos, numExceptions, vectorLen);
+ applyExceptionValues(pos, numExceptions);
+ }
+ }
+
+ /** Reads {@code numExceptions} little-endian uint16 positions into {@link #excPositionsBuffer}. */
+ private int readExceptionPositions(int pos, int numExceptions, int vectorLen) {
+ for (int e = 0; e < numExceptions; e++) {
+ excPositionsBuffer[e] = vectorsData.getShort(pos) & 0xFFFF;
Review Comment:
Added `BytesUtils.readUnsignedByte` and `readUnsignedShortLittleEndian`, with their own tests in `TestBytesUtil`, and switched the vector reads over to them. The short one reads little endian regardless of the buffer's byte order, the same as `readIntLittleEndian`. The 7-byte page header still uses relative reads, since absolute reads there broke when the page buffer didn't start at position 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-4850856207
I will try my best to find some time to review this large PR. That said, I don't think it should block the vote if interoperability has been verified across different PoCs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504193098
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
Review Comment:
Nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575350215
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpEncoderDecoderTest.java:
##########
@@ -0,0 +1,349 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import org.junit.Test;
+
+/**
+ * Tests for the core ALP encoder/decoder logic.
+ */
+public class AlpEncoderDecoderTest {
+
+ // ========== Float Encoding/Decoding Tests ==========
+
+ @Test
+ public void testFloatRoundTrip() {
+ float[] testValues = {0.0f, 1.0f, -1.0f, 3.14159f, 100.5f, 0.001f, 1234567.0f};
+
+ for (float value : testValues) {
+ for (int exponent = 0; exponent <= AlpConstants.FLOAT_MAX_EXPONENT; exponent++) {
+ for (int factor = 0; factor <= exponent; factor++) {
+ if (!AlpEncoderDecoder.isFloatException(value, exponent, factor)) {
+ int encoded = AlpEncoderDecoder.encodeFloat(value, exponent, factor);
+ float decoded = AlpEncoderDecoder.decodeFloat(encoded, exponent, factor);
+ assertEquals(
+ "Round-trip failed for value=" + value + ", exponent=" + exponent + ", factor="
+ + factor,
+ Float.floatToRawIntBits(value),
+ Float.floatToRawIntBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testFloatExceptionDetection() {
+ assertTrue("NaN should be an exception", AlpEncoderDecoder.isFloatException(Float.NaN));
+ assertTrue(
+ "Positive infinity should be an exception",
+ AlpEncoderDecoder.isFloatException(Float.POSITIVE_INFINITY));
+ assertTrue(
+ "Negative infinity should be an exception",
+ AlpEncoderDecoder.isFloatException(Float.NEGATIVE_INFINITY));
+ assertTrue("Negative zero should be an exception", AlpEncoderDecoder.isFloatException(-0.0f));
+
+ assertFalse("1.0f should not be a basic exception", AlpEncoderDecoder.isFloatException(1.0f));
+ assertFalse("0.0f should not be a basic exception", AlpEncoderDecoder.isFloatException(0.0f));
+ }
+
+ @Test
+ public void testFloatEncoding() {
+ assertEquals(123, AlpEncoderDecoder.encodeFloat(1.23f, 2, 0));
+ assertEquals(123, AlpEncoderDecoder.encodeFloat(12.3f, 2, 1));
+ assertEquals(0, AlpEncoderDecoder.encodeFloat(0.0f, 5, 0));
+ }
+
+ @Test
+ public void testFloatDecoding() {
+ assertEquals(1.23f, AlpEncoderDecoder.decodeFloat(123, 2, 0), 1e-6f);
+ assertEquals(12.3f, AlpEncoderDecoder.decodeFloat(123, 2, 1), 1e-6f);
+ assertEquals(0.0f, AlpEncoderDecoder.decodeFloat(0, 5, 0), 0.0f);
+ }
+
+ @Test
+ public void testFloatEncodeRounding() {
+ // Verify rounding behavior (magic number trick rounds to nearest)
+ assertEquals(5, AlpEncoderDecoder.encodeFloat(5.4f, 0, 0));
+ assertEquals(6, AlpEncoderDecoder.encodeFloat(5.6f, 0, 0));
+ assertEquals(-5, AlpEncoderDecoder.encodeFloat(-5.4f, 0, 0));
+ assertEquals(-6, AlpEncoderDecoder.encodeFloat(-5.6f, 0, 0));
+ assertEquals(0, AlpEncoderDecoder.encodeFloat(0.0f, 0, 0));
+ }
+
+ @Test
+ public void testFloatEncodeDecodeWithFactor() {
+ // Verify that encode/decode with non-zero factor works correctly.
+ // The key correctness property: encode uses value * POW10[e] * POW10_NEGATIVE[f],
+ // and decode uses encoded * POW10[f] * POW10_NEGATIVE[e].
+ float value = 12.3f;
+ int encoded = AlpEncoderDecoder.encodeFloat(value, 2, 1);
+ assertEquals(123, encoded); // 12.3 * 100 * 0.1 = 123
+ float decoded = AlpEncoderDecoder.decodeFloat(encoded, 2, 1);
+ assertEquals(Float.floatToRawIntBits(value), Float.floatToRawIntBits(decoded));
+ }
+
+ // ========== Double Encoding/Decoding Tests ==========
+
+ @Test
+ public void testDoubleRoundTrip() {
+ double[] testValues = {0.0, 1.0, -1.0, 3.14159265358979, 100.5, 0.001, 12345678901234.0};
+
+ for (double value : testValues) {
+ for (int exponent = 0; exponent <= Math.min(AlpConstants.DOUBLE_MAX_EXPONENT, 10); exponent++) {
+ for (int factor = 0; factor <= exponent; factor++) {
+ if (!AlpEncoderDecoder.isDoubleException(value, exponent, factor)) {
+ long encoded = AlpEncoderDecoder.encodeDouble(value, exponent, factor);
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, exponent, factor);
+ assertEquals(
+ "Round-trip failed for value=" + value + ", exponent=" + exponent + ", factor="
+ + factor,
+ Double.doubleToRawLongBits(value),
+ Double.doubleToRawLongBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testDoubleExceptionDetection() {
+ assertTrue("NaN should be an exception", AlpEncoderDecoder.isDoubleException(Double.NaN));
+ assertTrue(
+ "Positive infinity should be an exception",
+ AlpEncoderDecoder.isDoubleException(Double.POSITIVE_INFINITY));
+ assertTrue(
+ "Negative infinity should be an exception",
+ AlpEncoderDecoder.isDoubleException(Double.NEGATIVE_INFINITY));
+ assertTrue("Negative zero should be an exception", AlpEncoderDecoder.isDoubleException(-0.0));
+
+ assertFalse("1.0 should not be a basic exception", AlpEncoderDecoder.isDoubleException(1.0));
+ assertFalse("0.0 should not be a basic exception", AlpEncoderDecoder.isDoubleException(0.0));
+ }
+
+ @Test
+ public void testDoubleEncoding() {
+ assertEquals(123L, AlpEncoderDecoder.encodeDouble(1.23, 2, 0));
+ assertEquals(123L, AlpEncoderDecoder.encodeDouble(12.3, 2, 1));
+ assertEquals(0L, AlpEncoderDecoder.encodeDouble(0.0, 5, 0));
+ }
+
+ @Test
+ public void testDoubleDecoding() {
+ assertEquals(1.23, AlpEncoderDecoder.decodeDouble(123, 2, 0), 1e-10);
+ assertEquals(12.3, AlpEncoderDecoder.decodeDouble(123, 2, 1), 1e-10);
+ assertEquals(0.0, AlpEncoderDecoder.decodeDouble(0, 5, 0), 0.0);
+ }
+
+ @Test
+ public void testDoubleEncodeRounding() {
+ // Verify rounding behavior (magic number trick rounds to nearest)
+ assertEquals(5L, AlpEncoderDecoder.encodeDouble(5.4, 0, 0));
+ assertEquals(6L, AlpEncoderDecoder.encodeDouble(5.6, 0, 0));
+ assertEquals(-5L, AlpEncoderDecoder.encodeDouble(-5.4, 0, 0));
+ assertEquals(-6L, AlpEncoderDecoder.encodeDouble(-5.6, 0, 0));
+ assertEquals(0L, AlpEncoderDecoder.encodeDouble(0.0, 0, 0));
+ }
+
+ @Test
+ public void testDoubleEncodeDecodeWithFactor() {
+ // Verify that encode/decode with non-zero factor works correctly.
+ // The key correctness property: encode uses value * POW10[e] * POW10_NEGATIVE[f],
+ // and decode uses encoded * POW10[f] * POW10_NEGATIVE[e].
+ double value = 12.3;
+ long encoded = AlpEncoderDecoder.encodeDouble(value, 2, 1);
+ assertEquals(123L, encoded); // 12.3 * 100 * 0.1 = 123
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, 2, 1);
+ assertEquals(Double.doubleToRawLongBits(value), Double.doubleToRawLongBits(decoded));
+ }
+
+ @Test
+ public void testDoubleEncodeDecodeArithmeticOrder() {
+ // This test verifies that the exact order of operations in encode/decode
+ // is critical for IEEE 754 correctness. The encode uses
+ // fastRound(value * POW10[e] * POW10_NEGATIVE[f]) and decode uses
+ // (encoded * POW10[f] * POW10_NEGATIVE[e]), both as single expressions.
+ // Splitting the multiplies or reordering changes rounding.
+ double[] testValues = {0.123456789, 1.23456789, 12.3456789, 123.456789, 1234.56789};
+ for (double value : testValues) {
+ for (int e = 0; e <= 10; e++) {
+ for (int f = 0; f <= e; f++) {
+ if (!AlpEncoderDecoder.isDoubleException(value, e, f)) {
+ long encoded = AlpEncoderDecoder.encodeDouble(value, e, f);
+ double decoded = AlpEncoderDecoder.decodeDouble(encoded, e, f);
+ assertEquals(
+ "Roundtrip failed for " + value + " (e=" + e + ", f=" + f + ")",
+ Double.doubleToRawLongBits(value),
+ Double.doubleToRawLongBits(decoded));
+ }
+ }
+ }
+ }
+ }
+
+ // ========== Bit Width Tests (renamed methods) ==========
Review Comment:
bitWidthForLong isn't reimplementing an existing BytesUtils method, it's the long counterpart to getWidthFromMaxInt, which only handles ints. The javadoc on it says so now. This test just covers that helper.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpBitPackingTest.java:
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.junit.Test;
+
+/**
+ * Tests for bit-packing behavior in the ALP encoding pipeline.
+ */
+public class AlpBitPackingTest {
+
+ @Test
+ public void testBytePackerIntRoundTrip() {
+ // Verify BytePacker pack/unpack round-trip for various bit widths
+ for (int bitWidth = 1; bitWidth <= 31; bitWidth++) {
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(bitWidth);
+ int maxVal = (int) Math.min((1L << bitWidth) - 1, Integer.MAX_VALUE);
+
+ int[] input = new int[8];
+ for (int i = 0; i < 8; i++) {
+ input[i] = (maxVal / 8) * i;
+ }
+
+ byte[] packed = new byte[bitWidth];
+ packer.pack8Values(input, 0, packed, 0);
+
+ int[] unpacked = new int[8];
+ ByteBuffer buf = ByteBuffer.wrap(packed);
+ packer.unpack8Values(buf, 0, unpacked, 0);
+
+ for (int i = 0; i < 8; i++) {
+ assertEquals("BitWidth=" + bitWidth + " index=" + i, input[i], unpacked[i]);
+ }
+ }
+ }
+
+ @Test
+ public void testBytePackerForLongRoundTrip() {
+ // Verify BytePackerForLong pack/unpack round-trip for various bit widths
+ for (int bitWidth = 1; bitWidth <= 63; bitWidth++) {
Review Comment:
Extended the loops to the full width: int through 32 and long through 64, with the top bit of each width exercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3942046622
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
Review Comment:
Thanks for catching this, it's a good find. You're right that numElements is only bounded by valuesCount, and since a forged file controls that value it can go all the way up to Integer.MAX_VALUE. Two separate things went wrong from there. The `+ vectorSize - 1` overflowed into a negative count, and even when it didn't overflow, a large count drove a big allocation before anything had been validated.
While fixing it I found the allocation is genuinely reachable rather than theoretical. MultiBufferInputStream.slice() calls ByteBuffer.allocate(length) before it checks for EOF, so a small forged page really can ask for hundreds of megabytes.
This is fixed in 93c2ada1f. numVectors is computed in long now and checked against the bytes actually present before anything gets allocated. Since every vector needs a 4 byte offset entry plus at least its ALP and FOR headers, that gives a firm ceiling on how many vectors the remaining bytes could possibly describe, and anything above it throws a ParquetDecodingException that says how far short the page is.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
Review Comment:
You're right on all three counts, thank you. Fixed in 93c2ada1f.
initFromPage now validates the whole offset array before any vector is decoded. The first offset has to equal the offset array size, offsets have to increase by at least one vector's worth of fixed headers, and each one has to start early enough to leave room for those headers before the body ends.
The observation about decode ignoring the next offset was the more important half of this, since that's the case that quietly returns wrong values instead of failing. decodeVector now takes each vector's end position from the following offset, or from the end of the body for the last vector, and rejects a packed body or exception block that runs past it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ if (n < 0 || pageValueIndex + n > totalCount) {
Review Comment:
Good catch, thank you. Fixed in 93c2ada1f by comparing against the remaining count with `n > totalCount - pageValueIndex` rather than adding to pageValueIndex, so a large n can no longer overflow past the check and leave the reader at a negative index. I added testSkipRejectsOverflowingCount to cover it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
Review Comment:
Thank you for working through the arithmetic on this one, it turned out to be the most valuable comment in the review. I checked it and the numbers come out exactly as you described. At the default 20k row page limit with a 1024 vector size a page holds roughly 19 vectors, and rowgroupSampleJump works out to 15, so each page contributes about two samples. Because reset() cleared sampledParams and vectorsProcessed at every page boundary, the count never got near SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP, buildPresetCache() never ran, and findBest*ParamsWithPresets was effectively dead code. Every vector was paying for a full parameter search.
Fixed in 8f90f25fa. The sampler is meant to operate over a rowgroup rather than a page, so the sampling state now survives reset() and samples accumulate across pages the way the design intended. I added testPresetCacheBuildsAcrossPageBoundaries, which fails without the change.
One consequence I want to flag, since it isn't obvious. Now that the preset path is genuinely reachable, encoded output can shift slightly, because the search narrows to the top MAX_PRESET_COMBINATIONS pairs instead of trying everything. That is the intended ALP sampling behavior and it matches the C++ implementation. Round tripping is still exact, since the chosen pair is recorded in each vector's header and exceptions cover anything it cannot represent.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
+ Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[PACK_GROUP_SIZE];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Sampling phase first (full search + collect evenly-spaced samples, then build the preset
+ // cache once enough are gathered); after the cache is built, later vectors take the else branch.
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets == null) {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && sampledParams.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ sampledParams.add(new int[] {params.exponent, params.factor});
+ }
+ if (sampledParams.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
Review Comment:
This is the same underlying problem as your comment on the sampling cadence, and it's fixed in 8f90f25fa. I've written up the details in that thread rather than repeating them here. Thanks for flagging it from both angles, it made the problem much easier to pin down.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpValuesEndToEndTest.java:
##########
@@ -0,0 +1,2208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+/**
+ * End-to-end tests for ALP encoding and decoding pipeline.
+ */
+public class AlpValuesEndToEndTest {
+
+ private static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ // ========== Helper Methods ==========
+
+ /**
+ * Round-trip with STRICT raw-bit comparison for every value, including NaN payloads (the regular
+ * roundTrip helpers only assert isNaN, which would hide a NaN-payload-preservation bug). ALP must
+ * be losslessly bit-exact for every possible IEEE-754 bit pattern.
+ */
+ private void roundTripDoubleStrict(double[] values) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(512, values.length * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ long exp = Double.doubleToRawLongBits(values[i]);
+ long act = Double.doubleToRawLongBits(reader.readDouble());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Long.toHexString(exp)
+ + " actualBits=0x" + Long.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ private void roundTripFloatStrict(float[] values) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(256, values.length * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ int exp = Float.floatToRawIntBits(values[i]);
+ int act = Float.floatToRawIntBits(reader.readFloat());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Integer.toHexString(exp)
+ + " actualBits=0x" + Integer.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Full-bit-space fuzz (lossless invariant) ==========
+
+ @Test
+ public void testDoubleFullBitSpaceFuzz() throws Exception {
+ // Sweep the entire double bit space: random raw longs -> double. Covers all subnormals, every
+ // NaN payload, +/-0, +/-Inf, extreme exponents, chaotic mixed magnitudes within a vector.
+ Random rng = new Random(0x9E3779B97F4A7C15L);
+ for (int v = 0; v < 300; v++) { // ~300k values
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Double.longBitsToDouble(rng.nextLong());
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testFloatFullBitSpaceFuzz() throws Exception {
+ Random rng = new Random(0x9E3779B9L);
+ for (int v = 0; v < 300; v++) {
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Float.intBitsToFloat(rng.nextInt());
+ }
+ roundTripFloatStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleMixedFuzz() throws Exception {
+ // Mix ALP-friendly decimal-ish values with random raw bits, so FOR encoding AND the exception
+ // path are both exercised on chaotic data within the same vector.
+ Random rng = new Random(0xD1CE5EEDL);
+ for (int v = 0; v < 300; v++) {
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ if (rng.nextInt(4) == 0) {
+ values[i] = Double.longBitsToDouble(rng.nextLong()); // ~25% chaotic (mostly exceptions)
+ } else {
+ values[i] = Math.round(rng.nextDouble() * 1_000_000.0) / 100.0; // 2-decimal, ALP-friendly
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleDistributionShiftWithinRowGroup() throws Exception {
+ // First vectors are clean 2-decimal data (the sampler builds its preset (e,f) cache from these),
+ // then later vectors switch to high-precision / very different magnitude that the cached presets
+ // fit poorly. Once presets are cached the writer only tries those, so ALP must still stay lossless
+ // (the exception path catches every preset mismatch).
+ Random rng = new Random(7);
+ int totalVectors = 40; // well past SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP so presets are in use
+ double[] values = new double[totalVectors * 1024];
+ for (int i = 0; i < values.length; i++) {
+ int vec = i / 1024;
+ if (vec < 12) {
+ values[i] = Math.round(rng.nextDouble() * 10000.0) / 100.0; // clean cents
+ } else {
+ values[i] = rng.nextDouble() * 1e12 + rng.nextDouble(); // high-precision, big magnitude
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+
+ // ========== Reader robustness against malformed input ==========
+
+ private void readAllDoubles(byte[] bytes, int valueCount) throws Exception {
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(valueCount, ByteBufferInputStream.wrap(ByteBuffer.wrap(bytes)));
+ for (int i = 0; i < valueCount; i++) {
+ reader.readDouble();
+ }
+ }
+
+ @Test
+ @Timeout(value = 30, unit = TimeUnit.SECONDS)
+ public void testReaderRejectsCorruptInputCleanly() throws Exception {
+ // Build a valid ALP double page, then feed corrupted variants. The reader must fail cleanly
+ // (a catchable exception) and never hang, OOM, or read out of bounds silently.
+ double[] values = new double[2048];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (i % 100) / 100.0;
+ }
+ AlpValuesWriter.DoubleAlpValuesWriter writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ 65536, 65536, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ byte[] valid = writer.getBytes().toByteArray();
+ writer.reset();
+ writer.close();
+
+ // Sanity: the valid page reads fine.
+ readAllDoubles(valid, values.length);
+
+ // Corruptions that must each fail cleanly (never crash/hang/OOB):
+ java.util.List<byte[]> corrupt = new java.util.ArrayList<>();
+ corrupt.add(java.util.Arrays.copyOf(valid, valid.length / 2)); // truncated to half
+ corrupt.add(java.util.Arrays.copyOf(valid, 3)); // truncated to a stub
+ corrupt.add(new byte[0]); // empty
+ for (int pos : new int[] {0, 1, 2, 5, 7, 11, 20, valid.length - 1}) {
+ byte[] c = valid.clone();
+ c[pos] = (byte) ~c[pos]; // flip a header/body byte
+ corrupt.add(c);
+ }
+ for (byte[] c : corrupt) {
+ try {
+ readAllDoubles(c, values.length);
+ // Some single-byte flips may still decode to (wrong but in-bounds) values without throwing;
+ // that is acceptable here. What matters is no crash/hang/OOB, which we reached this line.
+ } catch (OutOfMemoryError oom) {
+ fail("Malformed input caused an OutOfMemoryError (allocation bomb) - a corrupt size/count "
+ + "must not drive an unbounded allocation");
+ } catch (Throwable t) {
+ // Any ordinary catchable exception (EOFException, ParquetDecodingException, IndexOutOfBounds,
+ // BufferUnderflow, NegativeArraySize, ...) is a clean failure: no JVM crash, no OOB, no hang.
+ }
+ }
+
+ // Claiming far more elements than the data supports must be rejected without an OOM allocation.
+ try {
+ readAllDoubles(valid, Integer.MAX_VALUE / 2);
Review Comment:
You're right, and thank you for reading the tests as carefully as the production code. That test only inflated the page's valuesCount, so the `numElements <= valuesCount` check passed trivially and the allocation path was never touched.
I replaced it in 93c2ada1f with testForgedElementCountIsRejectedWithoutHugeAllocation, which overwrites the num_elements field in the header directly and passes Integer.MAX_VALUE as valuesCount so that the allocation bound is what has to reject it. It covers both the counts that are simply too large for the page and the two that overflow the vector count arithmetic.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
+
+ // ---------------------------------------------------------------------------
+ // Helpers: build a known-good encoded page, then mutate copies of it
+ // ---------------------------------------------------------------------------
+
+ /** Build a valid ALP-encoded double page with N clean values. */
+ private static byte[] validDoublePage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(512, valueCount * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ // 2-decimal values — the ALP sweet spot, ensures no exceptions
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeDouble((i % 1000) / 100.0);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Build a valid ALP-encoded float page with N clean values. */
+ private static byte[] validFloatPage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(256, valueCount * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeFloat((i % 1000) / 100.0f);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Sanity baseline: the known-good page actually decodes cleanly. */
+ @Test
+ public void sanityBaselineDecodesClean() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Header-level validation (already-validated paths)
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsBadCompressionMode() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[0] = (byte) 0x99; // mode is at byte 0
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("compression"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBadIntegerEncoding() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[1] = (byte) 0x99; // integer_encoding is at byte 1
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("integer encoding"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooLarge() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 99; // log_vector_size at byte 2
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("vector size"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooSmall() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 2; // below MIN_LOG_VECTOR_SIZE=3
+ assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ }
+
+ @Test
+ public void rejectsNegativeNumElements() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements is int32 LE at bytes 3..6 — write -1
+ page[3] = (byte) 0xFF;
+ page[4] = (byte) 0xFF;
+ page[5] = (byte) 0xFF;
+ page[6] = (byte) 0xFF;
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("element count"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsNumElementsGreaterThanValuesCount() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements stays 32; pass valuesCount=10 (smaller than encoded count)
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(10, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("exceeds"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Vector-level validation (already-validated paths, surface lazily on decode)
+ // ---------------------------------------------------------------------------
+
+ /** Helper: find the byte position where the first vector's metadata starts. */
+ private static int firstVectorOffset(byte[] page) {
+ // header (7) + first 4 bytes of offset array = the offset value itself
+ int firstVectorOff =
+ ByteBuffer.wrap(page, 7, 4).order(ByteOrder.LITTLE_ENDIAN).getInt();
+ // offsets are measured from the start of the compression body (after the 7B header)
+ return 7 + firstVectorOff;
+ }
+
+ @Test
+ public void rejectsExponentTooHighDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99; // exponent byte
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsExponentTooHighFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsFactorGreaterThanExponent() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 2; // exponent
+ page[v0 + 1] = (byte) 5; // factor > exponent
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("factor"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsTooManyExceptions() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // num_exceptions at v0+2, uint16 LE — set to 9999, way more than vectorLen=16
+ page[v0 + 2] = (byte) (9999 & 0xFF);
+ page[v0 + 3] = (byte) ((9999 >>> 8) & 0xFF);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("numexceptions"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(8) then bitWidth byte at v0+12. 99 > 64.
+ page[v0 + 12] = (byte) 99;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(4) then bitWidth byte at v0+8. 99 > 32.
+ page[v0 + 8] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Currently-unvalidated paths: truncation and corrupted offsets
+ // These currently fail with low-level Throwables (BufferUnderflowException,
+ // IndexOutOfBoundsException). The tests assert any Throwable is raised so we
+ // notice if a regression silently swallows the corruption.
+ // ---------------------------------------------------------------------------
+
+ /** Page with only the 7-byte header — nothing else. */
+ @Test
+ public void rejectsHeaderOnlyPage() {
+ byte[] tiny = new byte[] {0x00, 0x00, 0x0A, 0x20, 0x00, 0x00, 0x00}; // 32 elements, log_vec=10
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(tiny)));
+ });
+ assertThat(t).as("header-only page must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidOffsetArray() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_vectors = ceil(32/16) = 2, so offset array is 8 bytes. Truncate to chop the 2nd offset.
+ byte[] truncated = new byte[7 + 4]; // header + first offset only
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ });
+ assertThat(t).as("truncated offset array must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidVectorData() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // chop the last 20 bytes — guaranteed to land in the middle of the second vector
+ byte[] truncated = new byte[page.length - 20];
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ // initFromPage should still succeed (truncation is inside the vectors section,
+ // which initFromPage just slices without parsing). The failure surfaces on decode.
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ Throwable t = catchAny(() -> {
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ });
+ assertThat(t).as("truncated vector data must raise on read").isNotNull();
+ }
+
+ @Test
+ public void rejectsCorruptedOffsetPointingPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // Offset array starts at byte 7. Overwrite the first offset (uint32 LE) with a huge value.
+ page[7] = (byte) 0xFF;
+ page[8] = (byte) 0xFF;
+ page[9] = (byte) 0xFF;
+ page[10] = (byte) 0x7F;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ Throwable t = catchAny(() -> reader.readDouble());
+ assertThat(t).as("corrupted offset must raise on decode").isNotNull();
+ }
+
+ // ---------------------------------------------------------------------------
+ // skip() / read() bounds
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsSkipPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(33));
+ }
+
+ @Test
+ public void rejectsNegativeSkip() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(-1));
+ }
+
+ @Test
+ public void rejectsReadPastEnd() throws Exception {
+ byte[] page = validDoublePage(8, 8);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(8, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 8; i++) reader.readDouble();
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exhausted"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Utility
+ // ---------------------------------------------------------------------------
+
+ @FunctionalInterface
+ private interface ThrowingRunnable {
+ void run() throws Throwable;
+ }
+
+ /** Catch any Throwable (including low-level RuntimeExceptions / Errors). */
+ private static Throwable catchAny(ThrowingRunnable r) {
Review Comment:
Agreed, and thanks for pointing it out. Catching Throwable meant the test could not really fail, which defeated the point of it.
In 93c2ada1f I replaced catchAny with catchClean, which catches Exception and asserts that the type is one of the expected clean failures while letting Error propagate. An OutOfMemoryError from an allocation bomb or an AssertionError from an inner assertion will now fail the test rather than quietly counting as a successful rejection. I applied the same change to the `catch (Throwable)` in the corruption loop in AlpValuesEndToEndTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3942060684
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +637,74 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Enable or disable ALP encoding for FLOAT and DOUBLE columns.
+ *
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(boolean enable) {
+ this.alpEnabled.withDefaultValue(enable);
+ return this;
+ }
+
+ /**
+ * Enable or disable ALP encoding for the specified column.
+ *
+ * @param columnPath the path of the column (dot-string)
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(String columnPath, boolean enable) {
+ this.alpEnabled.withValue(columnPath, enable);
+ return this;
+ }
+
+ /**
+ * Set the ALP vector size (number of values per encoded vector) for FLOAT and DOUBLE columns.
+ * Must be a power of 2 in the range supported by {@link AlpConstants}.
+ *
+ * @param vectorSize the vector size
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpVectorSize(int vectorSize) {
Review Comment:
You're right that this was more complicated than it needed to be, thanks for pushing on it. The builder was keeping the enabled flag and the vector size as two separate ColumnProperty scaffolds, merging them into an AlpConfig in buildAlp(), and then splitting them apart again in the copy constructor. Nothing actually needed that round trip.
Fixed in 68c103b84. There is now a single ColumnProperty<AlpConfig>, with withAlp(AlpConfig) and withAlp(columnPath, AlpConfig) as the primary API. I kept withAlpEncoding and withAlpVectorSize as thin read-modify-write conveniences on top, since they match the ergonomics of withByteStreamSplitEncoding and it means no existing caller had to change, including ParquetWriter.Builder. buildAlp and the copy constructor split are both gone.
I also moved the vector size validation into the AlpConfig constructor, which felt like the right home for it since an invalid size should be rejected however the config was built. That had the nice side effect of removing the last reference to AlpConstants from ParquetProperties, which made your visibility comment straightforward to act on.
One thing I want to flag for you. To let the convenience setters modify an existing config rather than replace it, I added getDefaultValue and getValue to ColumnProperty.Builder. That class is package private so it is not public API, but it is a shared file rather than something ALP specific, so please say if you would rather I found another way around it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
Review Comment:
Good question, and the answer is that it did not need to be public. I checked every reference and ParquetProperties was the only thing outside the package touching it, for DEFAULT_VECTOR_SIZE and validateVectorSize. Once the AlpConfig change above removed both of those, nothing external was left.
The class and all of its members are package private now, in 6872c2abd. It seemed worth doing properly before a release rather than after, since it keeps the ALP internals out of the public API surface permanently.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
+
+ private AlpConstants() {
+ // Utility class
+ }
+
+ // Page header fields
+ public static final int ALP_COMPRESSION_MODE = 0;
+ public static final int ALP_INTEGER_ENCODING_FOR = 0;
+ public static final int ALP_HEADER_SIZE = 7;
+
+ public static final int DEFAULT_VECTOR_SIZE = 1024;
+ public static final int DEFAULT_VECTOR_SIZE_LOG = 10;
+
+ // BytePacker packs/unpacks 8 values at a time (pack8Values/unpack8Values).
+ static final int PACK_GROUP_SIZE = 8;
+
+ // Capped at 15 (vectorSize=32768) because num_exceptions is uint16,
+ // so vectorSize must not exceed 65535 to avoid overflow when all values are exceptions.
+ static final int MAX_LOG_VECTOR_SIZE = 15;
+ static final int MIN_LOG_VECTOR_SIZE = 3;
+
+ static final int FLOAT_MAX_EXPONENT = 10;
+ static final int DOUBLE_MAX_EXPONENT = 18;
+
+ // Sampler constants matching C++ AlpConstants.
+ // Sample SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP vectors evenly distributed across a rowgroup
+ // of SAMPLER_ROWGROUP_SIZE values, then lock in top MAX_PRESET_COMBINATIONS combos.
+ static final int SAMPLER_ROWGROUP_SIZE = 122_880;
Review Comment:
Agreed, thank you. I went through each constant and sorted them by who actually reads them, in 6872c2abd.
The sampler constants (SAMPLER_ROWGROUP_SIZE, SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP, MAX_PRESET_COMBINATIONS) are only used by the writer, so they moved to AlpValuesWriter. The rounding magic numbers and the powers of ten tables are only used by the codec, so they moved to AlpCodec.
What is left in AlpConstants is genuinely shared and all of it describes the wire format: the header and metadata sizes, the mode and encoding markers, the vector size bounds, and the per type exponent limits. It reads much better as a wire format definition than it did as a general bucket.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
Review Comment:
Happy to take this. Renamed to AlpCodec in 6872c2abd, along with its test. I went with AlpCodec over AlpUtil since it really is a codec rather than a collection of helpers.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
Review Comment:
Understood, and sorry for the noise. All five files now use explicit static imports, in 6872c2abd. I did this last on purpose, after the constants had been moved to their new homes, so that the import lists reflect where things ended up rather than needing a second pass.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
Review Comment:
Thanks for spotting that. The cause is that the cell contains `4B × numVectors`, and the HTML entity is seven characters in the source but renders as a single glyph, so the box could never line up in both your editor and the rendered javadoc at the same time.
Fixed in 6872c2abd by using the literal character and padding the cell to the right width, so every row is 73 characters in both views. The same diagram had been copied into AlpValuesReader with the same problem, so I fixed it there too.
##########
parquet-format-structures/pom.xml:
##########
@@ -66,6 +66,35 @@
</execution>
</executions>
</plugin>
+ <!--
Review Comment:
Your timing on this was good. #3709 merged a couple of days after you left the comment, so parquet.thrift is now inlined in the repo and I was able to take the approach you suggested.
In 3bc0835bf, ALP = 10 is declared directly in the Encoding enum, and the perl script and the exec-maven-plugin execution that ran it are both gone. I verified the generated Encoding.java still carries ALP(10) after BYTE_STREAM_SPLIT(9) with no patch step involved.
There is one wrinkle I want your opinion on. dev/update-parquet-thrift.sh overwrites parquet.thrift wholesale from upstream, so the next time anyone runs it the ALP entry will disappear silently. It also rewrites the parquet-format.version sidecar, so a note there would not survive either. For now I have put a clearly marked notice in the enum itself saying it is a local addition that needs re-applying until ALP is accepted into parquet-format, on the grounds that at least the loss shows up in a diff. If you would prefer this handled a different way, such as a guard in the update script, I am glad to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3942850439
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
Review Comment:
Fair question, and the honest answer is that I was not making a deliberate choice here so much as following the layout that was already in the codebase. ByteStreamSplit is structured exactly the same way: ByteStreamSplitValuesWriter nests all five typed writers as public static classes, while its readers live in their own files as ByteStreamSplitValuesReaderForDouble, ByteStreamSplitValuesReaderForFloat and so on. ALP mirrors that, with the two writers nested and AlpValuesReaderForDouble and AlpValuesReaderForFloat separate.
So I agree the asymmetry is odd on its own terms, but it is the existing convention rather than something new, and matching the neighbouring encoding seemed more useful than being internally tidy in a way nothing else in the package is.
That said I do not feel strongly about it. If you would rather ALP be consistent within itself, splitting the two writers into their own files is a small and low risk change and I am happy to do it. It would just leave ALP looking different from ByteStreamSplit instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3997482649
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -415,7 +481,8 @@ public String toString() {
+ "Page row count limit to " + getPageRowCountLimit() + '\n'
+ "Writing page checksums is: " + (getPageWriteChecksumEnabled() ? "on" : "off") + '\n'
+ "Statistics enabled: " + statisticsEnabled + '\n'
- + "Size statistics enabled: " + sizeStatisticsEnabled;
+ + "Size statistics enabled: " + sizeStatisticsEnabled + '\n'
+ + "ALP: " + alp;
Review Comment:
Fixed in bdef58a44. It reads `ALP: off` when nothing is configured now, and `ALP: off {col_c=AlpConfig{vectorSize=1024}}` when only some columns have it, so the word null no longer appears either way.
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1406 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.values.alp.AlpConfig;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {"alp_float_spotify1.parquet", "alp_float_arade.parquet"};
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
+ if (dir == null) dir = System.getenv("ALP_TEST_DATA_DIR");
+ if (dir != null && new File(dir).isDirectory()) return Paths.get(dir);
+ // Default: alp-test-data/ relative to project root (two levels up from target/test-classes)
+ java.nio.file.Path candidate = Paths.get(System.getProperty("user.dir")).resolve("alp-test-data");
+ return candidate.toFile().isDirectory() ? candidate : null;
+ }
+
+ private java.nio.file.Path getSingleTestFile() {
+ String filePath = System.getProperty("ALP_TEST_FILE");
+ if (filePath == null) filePath = System.getenv("ALP_TEST_FILE");
+ if (filePath != null && new File(filePath).exists()) return Paths.get(filePath);
+ return null;
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ @Test
+ public void testReadSingleAlpFile() throws IOException {
+ java.nio.file.Path file = getSingleTestFile();
+ assumeTrue(file != null, "ALP_TEST_FILE not set or file does not exist, skipping");
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected at least one row in " + file).isTrue();
+ LOG.info("testReadSingleAlpFile: read {} rows from {}", rows.size(), file.getFileName());
+ }
+
+ @Test
+ public void testReadCppDoubleFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ for (String filename : CPP_DOUBLE_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue(file.toFile().exists(), "File not found: " + file);
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected rows in " + filename).isTrue();
+ LOG.info("testReadCppDoubleFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ @Test
+ public void testReadCppFloatFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ for (String filename : CPP_FLOAT_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue(file.toFile().exists(), "File not found: " + file);
+ List<Group> rows = readAllRows(file);
+ assertThat(rows.size() > 0).as("Expected rows in " + filename).isTrue();
+ LOG.info("testReadCppFloatFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ /** Verify no NaN/Inf corruption in float/double columns across all test files. */
+ @Test
+ public void testNoCorruptionInCppFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ String[] allFiles = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ int nanCount = 0;
+ int infCount = 0;
+ int totalValues = 0;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < pages.getRowCount(); i++) {
+ Group row = recordReader.read();
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ try {
+ if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float v = row.getFloat(f, 0);
+ totalValues++;
+ if (Float.isNaN(v)) nanCount++;
+ if (Float.isInfinite(v)) infCount++;
+ } else if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double v = row.getDouble(f, 0);
+ totalValues++;
+ if (Double.isNaN(v)) nanCount++;
+ if (Double.isInfinite(v)) infCount++;
+ }
+ } catch (Exception ignored) {
+ }
+ }
+ }
+ }
+ }
+ LOG.info("{}: {} values, {} NaN, {} Inf", filename, totalValues, nanCount, infCount);
+ assertThat(nanCount).as("Unexpected NaN in " + filename).isEqualTo(0);
+ assertThat(infCount).as("Unexpected Inf in " + filename).isEqualTo(0);
+ }
+ }
+
+ /** Schema inspection: log encoding types used in each column. */
+ @Test
+ public void testLogSchemaAndEncodings() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue(dir != null, "alp-test-data/ directory not found, skipping");
+ String[] allFiles = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ LOG.info("=== {} ===", filename);
+ LOG.info("Schema: {}", schema);
+ if (!footer.getBlocks().isEmpty()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col :
+ footer.getBlocks().get(0).getColumns()) {
+ LOG.info(
+ " column={} encodings={} compression={}",
+ col.getPath(),
+ col.getEncodings(),
+ col.getCodec());
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not read metadata for {}: {}", filename, e.getMessage());
+ }
+ }
+ // This test always passes — it's for inspection
+ assertThat(true).isTrue();
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { " + "required double double_col; " + "required float float_col; " + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlp()
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch for " + version).isEqualTo(DOUBLE_VALUES.length);
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i + " for " + version)
+ .isEqualTo(DOUBLE_VALUES[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i + " for " + version)
+ .isEqualTo(FLOAT_VALUES[i]);
+ }
+ LOG.info(
+ "writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}",
+ version,
+ rows.size(),
+ outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath = newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlp(new AlpConfig(4096))
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch at vectorSize=4096").isEqualTo(rowCount);
+ for (int i = 0; i < rowCount; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i)
+ .isEqualTo(doubles[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i)
+ .isEqualTo(floats[i]);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
Review Comment:
Guilty as charged, and the point lands — the patch it described was deleted in 229e0d99b, so the comment outlived the thing it documented by three commits. Dropped the detail rather than updating it in f7f7273aa.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098902712
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
Review Comment:
Total count of what? Value count?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099691327
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
+
+ // The packed body's width comes from the vector's own FOR header, so confirm it did not run
+ // past where the next vector begins; otherwise this vector would decode another one's bytes.
+ int vectorEnd = getVectorEndPosition(vectorNumber);
+ if (pos > vectorEnd) {
+ throw new ParquetDecodingException("ALP vector " + vectorNumber + " packed body ends at " + pos
+ + ", past the start of the next vector at " + vectorEnd);
+ }
+ if (numExceptions > 0) {
+ long exceptionBytes = (long) numExceptions * (Short.BYTES + exceptionValueSize());
+ if (pos + exceptionBytes > vectorEnd) {
+ throw new ParquetDecodingException(
+ "ALP vector " + vectorNumber + " declares " + numExceptions + " exceptions needing "
+ + exceptionBytes + " bytes, past the end of the vector at " + vectorEnd);
+ }
+ pos = readExceptionPositions(pos, numExceptions, vectorLen);
+ applyExceptionValues(pos, numExceptions);
+ }
+ }
+
+ /** Reads {@code numExceptions} little-endian uint16 positions into {@link #excPositionsBuffer}. */
+ private int readExceptionPositions(int pos, int numExceptions, int vectorLen) {
+ for (int e = 0; e < numExceptions; e++) {
+ excPositionsBuffer[e] = vectorsData.getShort(pos) & 0xFFFF;
Review Comment:
It would help to use a util method for `getShort(pos) & 0xFFFF`. That way you could have [tests that directly exercise it](https://github.com/apache/iceberg/blob/main/api/src/test/java/org/apache/iceberg/util/TestByteBuffers.java) and this would be more maintainable (you don't have to remember to mask, for example). See for
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
+
+ // The packed body's width comes from the vector's own FOR header, so confirm it did not run
+ // past where the next vector begins; otherwise this vector would decode another one's bytes.
+ int vectorEnd = getVectorEndPosition(vectorNumber);
+ if (pos > vectorEnd) {
+ throw new ParquetDecodingException("ALP vector " + vectorNumber + " packed body ends at " + pos
+ + ", past the start of the next vector at " + vectorEnd);
+ }
+ if (numExceptions > 0) {
+ long exceptionBytes = (long) numExceptions * (Short.BYTES + exceptionValueSize());
+ if (pos + exceptionBytes > vectorEnd) {
+ throw new ParquetDecodingException(
+ "ALP vector " + vectorNumber + " declares " + numExceptions + " exceptions needing "
+ + exceptionBytes + " bytes, past the end of the vector at " + vectorEnd);
+ }
+ pos = readExceptionPositions(pos, numExceptions, vectorLen);
+ applyExceptionValues(pos, numExceptions);
+ }
+ }
+
+ /** Reads {@code numExceptions} little-endian uint16 positions into {@link #excPositionsBuffer}. */
+ private int readExceptionPositions(int pos, int numExceptions, int vectorLen) {
+ for (int e = 0; e < numExceptions; e++) {
+ excPositionsBuffer[e] = vectorsData.getShort(pos) & 0xFFFF;
Review Comment:
It would help to use a util method for `getShort(pos) & 0xFFFF`. That way you could have [tests that directly exercise it](https://github.com/apache/iceberg/blob/main/api/src/test/java/org/apache/iceberg/util/TestByteBuffers.java) and this would be more maintainable (you don't have to remember to mask, for example).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3962579590
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +624,77 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Set the full ALP configuration for FLOAT and DOUBLE columns.
+ *
+ * @param config the ALP configuration
+ * @return this builder for method chaining.
+ */
+ public Builder withAlp(AlpConfig config) {
Review Comment:
if we did want the "off functions"
withoutAlp()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-4851365605
> That said, I don't think it should block the vote if interoperability has been verified across different PoCs.
I think someone needs to at least it won't require major rework, at least part of the existing guidelines say implementations are acceptable to maintainers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575359503
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpValuesEndToEndTest.java:
##########
@@ -0,0 +1,1745 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Random;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * End-to-end tests for ALP encoding and decoding pipeline.
+ */
+public class AlpValuesEndToEndTest {
+
+ private static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ // ========== Helper Methods ==========
+
+ private void roundTripFloat(float[] values) throws Exception {
+ roundTripFloat(values, DEFAULT_VECTOR_SIZE);
+ }
+
+ private void roundTripFloat(float[] values, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(256, values.length * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), vectorSize);
+
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float expected = values[i];
+ float actual = reader.readFloat();
+
+ if (Float.isNaN(expected)) {
+ assertTrue("Expected NaN at index " + i, Float.isNaN(actual));
+ } else {
+ assertEquals(
+ "Value mismatch at index " + i + " for " + expected,
+ Float.floatToRawIntBits(expected),
+ Float.floatToRawIntBits(actual));
+ }
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ private void roundTripDouble(double[] values) throws Exception {
+ roundTripDouble(values, DEFAULT_VECTOR_SIZE);
+ }
+
+ private void roundTripDouble(double[] values, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(512, values.length * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), vectorSize);
+
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ double expected = values[i];
+ double actual = reader.readDouble();
+
+ if (Double.isNaN(expected)) {
+ assertTrue("Expected NaN at index " + i, Double.isNaN(actual));
+ } else {
+ assertEquals(
+ "Value mismatch at index " + i + " for " + expected,
+ Double.doubleToRawLongBits(expected),
+ Double.doubleToRawLongBits(actual));
+ }
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Float Pipeline Tests ==========
+
+ @Test
+ public void testFloatSingleValue() throws Exception {
+ roundTripFloat(new float[] {1.23f});
+ }
+
+ @Test
+ public void testFloatSmallBatch() throws Exception {
+ roundTripFloat(new float[] {0.0f, 1.0f, -1.0f, 3.14f, 100.5f, 0.001f, 1234567.0f});
+ }
+
+ @Test
+ public void testFloatRandomData() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 10000.0f - 5000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatWithExceptions() throws Exception {
+ roundTripFloat(new float[] {
+ 1.0f, Float.NaN, 2.0f, Float.POSITIVE_INFINITY, 3.0f, Float.NEGATIVE_INFINITY, 4.0f, -0.0f, 5.0f
+ });
+ }
+
+ @Test
+ public void testFloatAllExceptions() throws Exception {
+ roundTripFloat(new float[] {Float.NaN, Float.POSITIVE_INFINITY, Float.NEGATIVE_INFINITY, -0.0f});
+ }
+
+ @Test
+ public void testFloatMultipleVectors() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[3000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 1000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatZeroValues() throws Exception {
+ roundTripFloat(new float[] {0.0f, 0.0f, 0.0f, 0.0f});
+ }
+
+ @Test
+ public void testFloatIdenticalValues() throws Exception {
+ float[] values = new float[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 3.14159f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Vector Size / Boundary Tests ==========
+
+ @Test
+ public void testFloatExactOneVector() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatExactTwoVectors() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 2];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatExactThreeVectors() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 3];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatPartialLastVectorPlusOne() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE + 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatPartialLastVectorMinusOne() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE * 2 - 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testFloatCustomVectorSize512() throws Exception {
+ float[] values = new float[512 + 100]; // partial second vector
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1f;
+ }
+ roundTripFloat(values, 512);
+ }
+
+ @Test
+ public void testFloatCustomVectorSize2048() throws Exception {
+ float[] values = new float[2048 * 2 + 500]; // partial third vector
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1f;
+ }
+ roundTripFloat(values, 2048);
+ }
+
+ @Test
+ public void testFloatCustomVectorSizeSmall() throws Exception {
+ // Minimum vector size = 8 (2^3)
+ float[] values = new float[25]; // 3 full vectors + 1 partial
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 1.5f;
+ }
+ roundTripFloat(values, 8);
+ }
+
+ // ========== Float: Large Page ==========
+
+ @Test
+ public void testFloatLargePage() throws Exception {
+ Random rand = new Random(42);
+ float[] values = new float[15000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 100000.0f - 50000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Monetary Data ==========
+
+ @Test
+ public void testFloatMonetaryData() throws Exception {
+ float[] values = {
+ 19.99f, 29.95f, 9.99f, 49.99f, 99.95f, 14.50f, 7.99f, 24.99f, 39.95f, 59.99f, 119.99f, 4.99f, 0.99f, 1.50f,
+ 2.99f, 3.49f
+ };
+ roundTripFloat(values);
+ }
+
+ // ========== Float: All Identical (bit_width = 0) ==========
+
+ @Test
+ public void testFloatAllIdenticalBitWidthZero() throws Exception {
+ float[] values = new float[2000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 42.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Mixed Exceptions at Various Positions ==========
+
+ @Test
+ public void testFloatExceptionsAtBoundaries() throws Exception {
+ // Exceptions at vector start and end
+ float[] values = new float[20];
+ values[0] = Float.NaN;
+ for (int i = 1; i < values.length - 1; i++) {
+ values[i] = i * 1.1f;
+ }
+ values[values.length - 1] = Float.POSITIVE_INFINITY;
+ roundTripFloat(values);
+ }
+
+ // ========== Float: Skip Tests ==========
+
+ @Test
+ public void testFloatSkip() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+
+ float[] values = {1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, 9.0f, 10.0f};
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ reader.skip(3);
+ assertEquals(Float.floatToRawIntBits(5.0f), Float.floatToRawIntBits(reader.readFloat()));
+ reader.skip(2);
+ assertEquals(Float.floatToRawIntBits(8.0f), Float.floatToRawIntBits(reader.readFloat()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test
+ public void testFloatSkipAcrossVectorBoundaries() throws Exception {
+ int vectorSize = 8; // small vector for boundary testing
+ int totalValues = vectorSize * 3 + 4; // 3 full vectors + partial
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ totalValues * 8, totalValues * 8, new DirectByteBufferAllocator(), vectorSize);
+
+ float[] values = new float[totalValues];
+ for (int i = 0; i < totalValues; i++) {
+ values[i] = (i + 1) * 1.0f;
+ writer.writeFloat(values[i]);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(totalValues, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ // Read first value from vector 0
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+
+ // Skip past vector 0 and vector 1, into vector 2
+ reader.skip(vectorSize * 2 - 1); // skip to index vectorSize*2
+
+ // Read first value of vector 2
+ float expected = (vectorSize * 2 + 1) * 1.0f;
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+
+ // Skip into the partial last vector
+ reader.skip(vectorSize - 1); // skip to index vectorSize*3
+
+ expected = (vectorSize * 3 + 1) * 1.0f;
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Float: Writer Reset ==========
+
+ @Test
+ public void testFloatWriterReset() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+ writer.reset();
+ assertEquals(0, writer.getBufferedSize());
+
+ float[] values = {3.0f, 4.0f, 5.0f};
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (float expected : values) {
+ assertEquals(Float.floatToRawIntBits(expected), Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Double Pipeline Tests ==========
+
+ @Test
+ public void testDoubleSingleValue() throws Exception {
+ roundTripDouble(new double[] {1.23456789});
+ }
+
+ @Test
+ public void testDoubleSmallBatch() throws Exception {
+ roundTripDouble(new double[] {0.0, 1.0, -1.0, 3.14159265358979, 100.5, 0.001, 12345678901234.0});
+ }
+
+ @Test
+ public void testDoubleRandomData() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 1000000.0 - 500000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleWithExceptions() throws Exception {
+ roundTripDouble(new double[] {
+ 1.0, Double.NaN, 2.0, Double.POSITIVE_INFINITY, 3.0, Double.NEGATIVE_INFINITY, 4.0, -0.0, 5.0
+ });
+ }
+
+ @Test
+ public void testDoubleAllExceptions() throws Exception {
+ roundTripDouble(new double[] {Double.NaN, Double.POSITIVE_INFINITY, Double.NEGATIVE_INFINITY, -0.0});
+ }
+
+ @Test
+ public void testDoubleMultipleVectors() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[3000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 100000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleZeroValues() throws Exception {
+ roundTripDouble(new double[] {0.0, 0.0, 0.0, 0.0});
+ }
+
+ @Test
+ public void testDoubleIdenticalValues() throws Exception {
+ double[] values = new double[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 3.14159265358979;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Vector Size / Boundary Tests ==========
+
+ @Test
+ public void testDoubleExactOneVector() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleExactTwoVectors() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE * 2];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoublePartialLastVectorPlusOne() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE + 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoublePartialLastVectorMinusOne() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE * 2 - 1];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.01;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSize512() throws Exception {
+ double[] values = new double[512 + 100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1;
+ }
+ roundTripDouble(values, 512);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSize2048() throws Exception {
+ double[] values = new double[2048 * 2 + 500];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 0.1;
+ }
+ roundTripDouble(values, 2048);
+ }
+
+ @Test
+ public void testDoubleCustomVectorSizeSmall() throws Exception {
+ double[] values = new double[25];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = i * 1.5;
+ }
+ roundTripDouble(values, 8);
+ }
+
+ // ========== Double: Large Page ==========
+
+ @Test
+ public void testDoubleLargePage() throws Exception {
+ Random rand = new Random(42);
+ double[] values = new double[15000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 100000.0 - 50000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Monetary Data ==========
+
+ @Test
+ public void testDoubleMonetaryData() throws Exception {
+ double[] values = {
+ 19.99, 29.95, 9.99, 49.99, 99.95, 14.50, 7.99, 24.99, 39.95, 59.99, 119.99, 4.99, 0.99, 1.50, 2.99, 3.49
+ };
+ roundTripDouble(values);
+ }
+
+ // ========== Double: All Identical (bit_width = 0) ==========
+
+ @Test
+ public void testDoubleAllIdenticalBitWidthZero() throws Exception {
+ double[] values = new double[2000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 42.0;
+ }
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Mixed Exceptions at Boundaries ==========
+
+ @Test
+ public void testDoubleExceptionsAtBoundaries() throws Exception {
+ double[] values = new double[20];
+ values[0] = Double.NaN;
+ for (int i = 1; i < values.length - 1; i++) {
+ values[i] = i * 1.1;
+ }
+ values[values.length - 1] = Double.POSITIVE_INFINITY;
+ roundTripDouble(values);
+ }
+
+ // ========== Double: Skip Tests ==========
+
+ @Test
+ public void testDoubleSkip() throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+
+ double[] values = {1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0};
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Double.doubleToRawLongBits(1.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(3);
+ assertEquals(Double.doubleToRawLongBits(5.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(2);
+ assertEquals(Double.doubleToRawLongBits(8.0), Double.doubleToRawLongBits(reader.readDouble()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test
+ public void testDoubleSkipAcrossVectorBoundaries() throws Exception {
+ int vectorSize = 8;
+ int totalValues = vectorSize * 3 + 4;
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ totalValues * 16, totalValues * 16, new DirectByteBufferAllocator(), vectorSize);
+
+ double[] values = new double[totalValues];
+ for (int i = 0; i < totalValues; i++) {
+ values[i] = (i + 1) * 1.0;
+ writer.writeDouble(values[i]);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(totalValues, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ assertEquals(Double.doubleToRawLongBits(1.0), Double.doubleToRawLongBits(reader.readDouble()));
+ reader.skip(vectorSize * 2 - 1);
+
+ double expected = (vectorSize * 2 + 1) * 1.0;
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+
+ reader.skip(vectorSize - 1);
+ expected = (vectorSize * 3 + 1) * 1.0;
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Double: Writer Reset ==========
+
+ @Test
+ public void testDoubleWriterReset() throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+
+ writer.writeDouble(1.0);
+ writer.writeDouble(2.0);
+ writer.reset();
+ assertEquals(0, writer.getBufferedSize());
+
+ double[] values = {3.0, 4.0, 5.0};
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (double expected : values) {
+ assertEquals(Double.doubleToRawLongBits(expected), Double.doubleToRawLongBits(reader.readDouble()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Packed Size Verification ==========
+
+ @Test
+ public void testPackedSizeSpecFormula() throws Exception {
+ // Verify that the packed data size exactly matches ceil(n * bitWidth / 8)
+ // by checking total encoded bytes for partial vectors with various remainders.
+ // Vector size 8 means: 1 full group of 8 = no partial group,
+ // but with count not divisible by 8 we exercise partial groups.
+ for (int count : new int[] {1, 2, 3, 5, 7, 9, 11, 13, 15}) {
+ float[] values = new float[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 1.0f;
+ }
+ // Use smallest vector size so the partial vector IS the only vector
+ roundTripFloat(values, 1024);
+ }
+ }
+
+ @Test
+ public void testPartialVectorAllRemaindersMod8() throws Exception {
+ // Test every possible remainder (1..7) to exercise the partial group path.
+ // With vectorSize=16: 16 is 2 full groups of 8, no partial group.
+ // So use vectorSize=16 and write 16+R values where R in {1..7}
+ // to get a partial last vector of size R.
+ for (int remainder = 1; remainder <= 7; remainder++) {
+ int count = 16 + remainder;
+ float[] values = new float[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 0.5f;
+ }
+ roundTripFloat(values, 16);
+ }
+ // Also test for doubles
+ for (int remainder = 1; remainder <= 7; remainder++) {
+ int count = 16 + remainder;
+ double[] values = new double[count];
+ for (int i = 0; i < count; i++) {
+ values[i] = (i + 1) * 0.5;
+ }
+ roundTripDouble(values, 16);
+ }
+ }
+
+ // ========== Negative Encoded Values / Wide FOR Range ==========
+
+ @Test
+ public void testFloatNegativeValues() throws Exception {
+ // Values that produce negative encoded integers, testing FOR with negative min
+ float[] values = new float[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = -50.0f + i * 1.0f; // range [-50, 49]
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleNegativeValues() throws Exception {
+ double[] values = new double[100];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = -50.0 + i * 1.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatWideRange() throws Exception {
+ // Mix of small and large positive values - exercises higher bit widths
+ float[] values = {0.01f, 0.02f, 1000.0f, 2000.0f, 0.05f, 50000.0f, 0.99f, 99999.0f};
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleWideRange() throws Exception {
+ double[] values = {0.01, 0.02, 100000.0, 200000.0, 0.05, 5000000.0, 0.99, 99999999.0};
+ roundTripDouble(values);
+ }
+
+ // ========== Binary Format Verification ==========
+
+ @Test
+ public void testBinaryFormatExactBytes() throws Exception {
+ // Write a known small dataset with vectorSize=8, then verify the
+ // exact binary layout byte-by-byte
+ int vectorSize = 8;
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), vectorSize);
+
+ // Write 10 identical values: 1.0f
+ // With e=0, f=0: encoded=1, FOR min=1, all deltas=0, bitWidth=0
+ for (int i = 0; i < 10; i++) {
+ writer.writeFloat(1.0f);
+ }
+
+ BytesInput input = writer.getBytes();
+ byte[] bytes = input.toByteArray();
+ java.nio.ByteBuffer buf = java.nio.ByteBuffer.wrap(bytes).order(java.nio.ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ assertEquals("compression_mode", 0, buf.get() & 0xFF);
+ assertEquals("integer_encoding", 0, buf.get() & 0xFF);
+ assertEquals("log_vector_size", 3, buf.get() & 0xFF); // log2(8)=3
+ assertEquals("num_elements", 10, buf.getInt());
+
+ // Offset array (2 vectors * 4 bytes = 8 bytes)
+ int offset0 = buf.getInt(); // first vector offset
+ int offset1 = buf.getInt(); // second vector offset
+ assertEquals("offset0 = past offset array", 8, offset0);
+
+ // Vector 0: 8 identical values, bitWidth=0, no exceptions
+ // AlpInfo (4 bytes)
+ int v0Exp = buf.get() & 0xFF;
+ int v0Fac = buf.get() & 0xFF;
+ int v0Exc = buf.getShort() & 0xFFFF;
+ assertEquals("v0 exceptions", 0, v0Exc);
+
+ // ForInfo (5 bytes)
+ int v0For = buf.getInt();
+ int v0Bw = buf.get() & 0xFF;
+ assertEquals("v0 bitWidth", 0, v0Bw);
+ // No packed bytes (bitWidth=0)
+ // No exceptions
+
+ // Vector 0 should be exactly 4 + 5 = 9 bytes
+ assertEquals("offset1 should be offset0 + 9", offset0 + 9, offset1);
+
+ // Vector 1: 2 identical values, bitWidth=0, no exceptions
+ int v1Exp = buf.get() & 0xFF;
+ int v1Fac = buf.get() & 0xFF;
+ assertEquals("same exponent both vectors", v0Exp, v1Exp);
+ assertEquals("same factor both vectors", v0Fac, v1Fac);
+ int v1Exc = buf.getShort() & 0xFFFF;
+ assertEquals("v1 exceptions", 0, v1Exc);
+ int v1For = buf.getInt();
+ int v1Bw = buf.get() & 0xFF;
+ assertEquals("v1 bitWidth", 0, v1Bw);
+
+ // Should have consumed all bytes
+ assertEquals("all bytes consumed", bytes.length, buf.position());
+
+ // Verify round-trip
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(10, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < 10; i++) {
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Exception-Heavy Vectors ==========
+
+ @Test
+ public void testFloatManyExceptionsInVector() throws Exception {
+ // Vector with more exceptions than normal values
+ float[] values = new float[10];
+ values[0] = 1.0f; // normal
+ for (int i = 1; i < 10; i++) {
+ values[i] = Float.NaN; // all exceptions except first
+ }
+ roundTripFloat(values, 16);
+ }
+
+ @Test
+ public void testDoubleManyExceptionsInVector() throws Exception {
+ double[] values = new double[10];
+ values[0] = 1.0;
+ for (int i = 1; i < 10; i++) {
+ values[i] = Double.NaN;
+ }
+ roundTripDouble(values, 16);
+ }
+
+ @Test
+ public void testFloatExceptionsAcrossMultipleVectors() throws Exception {
+ // Exceptions in different vectors
+ int vectorSize = 8;
+ float[] values = new float[vectorSize * 3];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (i + 1) * 0.5f;
+ }
+ // Place exceptions at specific positions in different vectors
+ values[0] = Float.NaN; // vector 0, position 0
+ values[vectorSize - 1] = Float.POSITIVE_INFINITY; // vector 0, last position
+ values[vectorSize] = -0.0f; // vector 1, position 0
+ values[vectorSize * 2 + 3] = Float.NaN; // vector 2, position 3
+ roundTripFloat(values, vectorSize);
+ }
+
+ // ========== Read After Skip to End ==========
+
+ @Test(expected = ParquetDecodingException.class)
+ public void testFloatReadPastEnd() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ writer.writeFloat(1.0f);
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(1, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ reader.readFloat(); // read the one value
+ reader.readFloat(); // should throw
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ @Test(expected = ParquetDecodingException.class)
+ public void testFloatSkipPastEnd() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(2, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ reader.skip(3); // should throw - only 2 values
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Hand-Crafted Binary → Reader (independent of writer) ==========
+
+ /**
+ * Construct ALP-encoded bytes manually per the spec and verify the reader
+ * decodes them correctly. This catches symmetric bugs where writer and reader
+ * both implement the same wrong thing.
+ *
+ * <p>Test vector: 3 float values {100.0f, 200.0f, 300.0f} with vectorSize=8.
+ * With exponent=0, factor=0: encoded = {100, 200, 300}.
+ * FOR min = 100, deltas = {0, 100, 200}, maxDelta = 200, bitWidth = 8.
+ * Packed: 3 values in 1 partial group of 8, padded with zeros.
+ * Packed size = ceil(3 * 8 / 8) = 3 bytes.
+ */
+ @Test
+ public void testReaderWithHandCraftedBytes() throws Exception {
+ // Manually build ALP page for {100.0f, 200.0f, 300.0f}, vectorSize=8
+ int vectorSize = 8;
+ int logVectorSize = 3;
+ int numElements = 3;
+ int numVectors = 1;
+
+ // Encoding: e=0, f=0 → multiplier=1.0
+ // encoded = {100, 200, 300}
+ // FOR min = 100
+ // deltas = {0, 100, 200}
+ // maxDelta = 200, bitWidth = 8
+ int exponent = 0;
+ int factor = 0;
+ int numExceptions = 0;
+ int frameOfReference = 100;
+ int bitWidth = 8;
+
+ // Bit-pack deltas using the same BytePacker the reader will use
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(8);
+ int[] deltas = {0, 100, 200, 0, 0, 0, 0, 0}; // padded to 8
+ byte[] packed = new byte[8]; // bitWidth bytes for one group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 8 + 7) / 8; // = 3 bytes for 3 values at 8 bits
+
+ // Build the page
+ int vectorDataSize = 4 + 5 + packedSize; // AlpInfo + ForInfo + packed
+ int offsetArraySize = numVectors * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) logVectorSize);
+ page.putInt(numElements);
+
+ // Offset array (1 vector)
+ page.putInt(offsetArraySize); // offset₀ = past offset array
+
+ // Vector 0: AlpInfo (4 bytes)
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) numExceptions);
+
+ // Vector 0: ForInfo (5 bytes)
+ page.putInt(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Vector 0: Packed data (3 bytes)
+ page.put(packed, 0, packedSize);
+
+ page.flip();
+
+ // Now read it back
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Float.floatToRawIntBits(100.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(200.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(300.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ /**
+ * Hand-craft bytes with exceptions to verify reader handles them independently.
+ * Values: {1.0f, NaN, 3.0f} with vectorSize=8.
+ * e=0, f=0: encoded={1, placeholder=1, 3}, 1 exception at position 1.
+ * FOR min=1, deltas={0, 0, 2}, maxDelta=2, bitWidth=2.
+ */
+ @Test
+ public void testReaderWithHandCraftedExceptions() throws Exception {
+ int vectorSize = 8;
+ int logVectorSize = 3;
+ int numElements = 3;
+
+ int exponent = 0;
+ int factor = 0;
+ int numExceptions = 1;
+ int placeholder = 1; // first non-exception encoded value
+ int frameOfReference = 1; // min of {1, 1, 3}
+ int bitWidth = 2; // ceil(log2(2+1)) = 2
+
+ // deltas = {1-1, 1-1, 3-1} = {0, 0, 2}, padded to 8: {0,0,2,0,0,0,0,0}
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(2);
+ int[] deltas = {0, 0, 2, 0, 0, 0, 0, 0};
+ byte[] packed = new byte[2]; // bitWidth=2 bytes for a group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 2 + 7) / 8; // = 1 byte
+
+ // Vector data: AlpInfo(4) + ForInfo(5) + packed(1) + excPos(2) + excVal(4) = 16
+ int vectorDataSize = 4 + 5 + packedSize + 2 + 4;
+ int offsetArraySize = 1 * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) logVectorSize);
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize);
+
+ // AlpInfo
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) numExceptions);
+
+ // ForInfo
+ page.putInt(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Packed data
+ page.put(packed, 0, packedSize);
+
+ // Exception positions (uint16 LE)
+ page.putShort((short) 1); // position 1
+
+ // Exception values (float32 LE)
+ page.putFloat(Float.NaN);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Float.floatToRawIntBits(1.0f), Float.floatToRawIntBits(reader.readFloat()));
+ float val1 = reader.readFloat();
+ assertTrue("Expected NaN at position 1", Float.isNaN(val1));
+ assertEquals(Float.floatToRawIntBits(3.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ /**
+ * Hand-craft double ALP bytes and verify reader independence.
+ * Values: {10.0, 20.0, 30.0}, vectorSize=8, e=0, f=0.
+ * encoded={10,20,30}, FOR min=10, deltas={0,10,20}, maxDelta=20, bitWidth=5.
+ */
+ @Test
+ public void testDoubleReaderWithHandCraftedBytes() throws Exception {
+ int numElements = 3;
+ int exponent = 0;
+ int factor = 0;
+ long frameOfReference = 10;
+ int bitWidth = 5; // ceil(log2(20+1)) = 5
+
+ org.apache.parquet.column.values.bitpacking.BytePackerForLong packer =
+ Packer.LITTLE_ENDIAN.newBytePackerForLong(5);
+ long[] deltas = {0, 10, 20, 0, 0, 0, 0, 0};
+ byte[] packed = new byte[5]; // bitWidth bytes for group of 8
+ packer.pack8Values(deltas, 0, packed, 0);
+ int packedSize = (3 * 5 + 7) / 8; // = 2 bytes
+
+ int vectorDataSize = 4 + 9 + packedSize; // AlpInfo + DoubleForInfo + packed
+ int offsetArraySize = 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + vectorDataSize).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) 3); // log2(8)
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize);
+
+ // AlpInfo
+ page.put((byte) exponent);
+ page.put((byte) factor);
+ page.putShort((short) 0); // no exceptions
+
+ // ForInfo (9 bytes for double)
+ page.putLong(frameOfReference);
+ page.put((byte) bitWidth);
+
+ // Packed data
+ page.put(packed, 0, packedSize);
+
+ page.flip();
+
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ assertEquals(Double.doubleToRawLongBits(10.0), Double.doubleToRawLongBits(reader.readDouble()));
+ assertEquals(Double.doubleToRawLongBits(20.0), Double.doubleToRawLongBits(reader.readDouble()));
+ assertEquals(Double.doubleToRawLongBits(30.0), Double.doubleToRawLongBits(reader.readDouble()));
+ }
+
+ // ========== Writer Output → Hand-Verified Bytes ==========
+
+ /**
+ * Write known values, then verify the exact binary output against hand computation.
+ * This verifies the writer independently of the reader.
+ *
+ * <p>Input: {1.0f, 2.0f, 3.0f} with vectorSize=8.
+ * Expected: e=0, f=0, encoded={1,2,3}, FOR min=1, deltas={0,1,2},
+ * maxDelta=2, bitWidth=2. Packed size = ceil(3*2/8) = 1 byte.
+ */
+ @Test
+ public void testWriterOutputExactBytes() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 8);
+ writer.writeFloat(1.0f);
+ writer.writeFloat(2.0f);
+ writer.writeFloat(3.0f);
+
+ byte[] bytes = writer.getBytes().toByteArray();
+ ByteBuffer buf = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ assertEquals(0, buf.get() & 0xFF); // compression_mode
+ assertEquals(0, buf.get() & 0xFF); // integer_encoding
+ assertEquals(3, buf.get() & 0xFF); // log2(8) = 3
+ assertEquals(3, buf.getInt()); // num_elements
+
+ // Offset array (1 vector)
+ int offset0 = buf.getInt();
+ assertEquals(4, offset0); // 1 * 4 = past offset array
+
+ // AlpInfo
+ int exp = buf.get() & 0xFF;
+ int fac = buf.get() & 0xFF;
+ int numExc = buf.getShort() & 0xFFFF;
+ assertEquals(0, numExc);
+
+ // ForInfo
+ int forRef = buf.getInt();
+ int bw = buf.get() & 0xFF;
+
+ // Verify: for e=0, f=0: multiplier=1.0
+ // encoded = {1, 2, 3}, min = 1, deltas = {0, 1, 2}, bitWidth = 2
+ assertEquals(1, forRef); // frame of reference = 1
+ assertEquals(2, bw); // bitWidth for max delta 2
+
+ // Packed data: ceil(3 * 2 / 8) = 1 byte
+ // deltas {0, 1, 2} at bitWidth=2:
+ // Use BytePacker to compute expected packed byte
+ BytePacker packer = Packer.LITTLE_ENDIAN.newBytePacker(2);
+ int[] expectedDeltas = {0, 1, 2, 0, 0, 0, 0, 0};
+ byte[] expectedPacked = new byte[2]; // bitWidth=2
+ packer.pack8Values(expectedDeltas, 0, expectedPacked, 0);
+
+ byte actualPackedByte = buf.get();
+ assertEquals("packed byte", expectedPacked[0], actualPackedByte);
+
+ // Should have consumed all bytes
+ assertEquals("all bytes consumed", bytes.length, buf.position());
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== NaN Bit Pattern Preservation ==========
+
+ /**
+ * Verify that different NaN bit patterns survive the round-trip exactly.
+ * Java allows multiple NaN representations. Our exception handling must
+ * preserve the exact bit pattern, not normalize to Float.NaN.
+ */
+ @Test
+ public void testNaNBitPatternPreservation() throws Exception {
+ // Standard NaN
+ float standardNaN = Float.NaN; // 0x7FC00000
+ // Signaling NaN (different bit pattern)
+ float signalingNaN = Float.intBitsToFloat(0x7F800001);
+ // Negative NaN
+ float negativeNaN = Float.intBitsToFloat(0xFFC00000);
+ // Another NaN variant
+ float customNaN = Float.intBitsToFloat(0x7FFFFFFF);
+
+ assertTrue(Float.isNaN(standardNaN));
+ assertTrue(Float.isNaN(signalingNaN));
+ assertTrue(Float.isNaN(negativeNaN));
+ assertTrue(Float.isNaN(customNaN));
+
+ float[] values = {1.0f, standardNaN, 2.0f, signalingNaN, 3.0f, negativeNaN, 4.0f, customNaN};
+
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float actual = reader.readFloat();
+ assertEquals(
+ "Bit pattern mismatch at index " + i + " (0x"
+ + Integer.toHexString(Float.floatToRawIntBits(values[i])) + " vs 0x"
+ + Integer.toHexString(Float.floatToRawIntBits(actual)) + ")",
+ Float.floatToRawIntBits(values[i]),
+ Float.floatToRawIntBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /**
+ * Same test for double NaN bit patterns.
+ */
+ @Test
+ public void testDoubleNaNBitPatternPreservation() throws Exception {
+ double standardNaN = Double.NaN;
+ double signalingNaN = Double.longBitsToDouble(0x7FF0000000000001L);
+ double negativeNaN = Double.longBitsToDouble(0xFFF8000000000000L);
+ double customNaN = Double.longBitsToDouble(0x7FFFFFFFFFFFFFFFL);
+
+ double[] values = {1.0, standardNaN, 2.0, signalingNaN, 3.0, negativeNaN, 4.0, customNaN};
+
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(512, 512, new DirectByteBufferAllocator());
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ double actual = reader.readDouble();
+ assertEquals(
+ "Bit pattern mismatch at index " + i + " (0x"
+ + Long.toHexString(Double.doubleToRawLongBits(values[i])) + " vs 0x"
+ + Long.toHexString(Double.doubleToRawLongBits(actual)) + ")",
+ Double.doubleToRawLongBits(values[i]),
+ Double.doubleToRawLongBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Negative Zero Bit-Exact Roundtrip ==========
+
+ /**
+ * Verify that -0.0f roundtrips as -0.0f (bit pattern 0x80000000),
+ * not as +0.0f (bit pattern 0x00000000).
+ */
+ @Test
+ public void testNegativeZeroBitExact() throws Exception {
+ float negZero = -0.0f;
+ float posZero = 0.0f;
+
+ // Sanity: they are == but have different bit patterns
+ assertTrue(negZero == posZero);
+ assertNotEquals(Float.floatToRawIntBits(negZero), Float.floatToRawIntBits(posZero));
+
+ float[] values = {posZero, negZero, 1.0f, negZero};
+
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator());
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < values.length; i++) {
+ float actual = reader.readFloat();
+ assertEquals(
+ "Bit pattern at index " + i,
+ Float.floatToRawIntBits(values[i]),
+ Float.floatToRawIntBits(actual));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Extreme Float Values ==========
+
+ /**
+ * Test subnormal floats, Float.MIN_VALUE, Float.MAX_VALUE, Float.MIN_NORMAL.
+ * These should all either encode losslessly or be stored as exceptions.
+ */
+ @Test
+ public void testExtremeFloatValues() throws Exception {
+ float[] values = {
+ Float.MIN_VALUE, // smallest positive subnormal: 1.4e-45
+ Float.MIN_NORMAL, // smallest positive normal: 1.17549435e-38
+ Float.MAX_VALUE, // 3.4028235e38
+ -Float.MAX_VALUE, // most negative
+ 1.17549435e-38f, // near MIN_NORMAL
+ 3.4028234e38f, // near MAX_VALUE
+ 1.0e-10f, // very small positive
+ 1.0e10f, // large positive
+ };
+ roundTripFloat(values);
+ }
+
+ /**
+ * Test extreme double values.
+ */
+ @Test
+ public void testExtremeDoubleValues() throws Exception {
+ double[] values = {
+ Double.MIN_VALUE, // smallest positive subnormal
+ Double.MIN_NORMAL, // smallest positive normal
+ Double.MAX_VALUE, // largest positive
+ -Double.MAX_VALUE, // most negative
+ 1.0e-100, // very small
+ 1.0e100, // very large
+ };
+ roundTripDouble(values);
+ }
+
+ // ========== Preset Caching Under Distribution Change ==========
+
+ /**
+ * Write >8 vectors where the optimal (e,f) changes after the sampling phase.
+ * First 8 vectors: monetary data (e=2, f=0 optimal).
+ * Remaining vectors: large integers (e=0, f=0 optimal).
+ * Verify all values survive round-trip despite the distribution change.
+ */
+ @Test
+ public void testPresetCachingWithDistributionChange() throws Exception {
+ int vectorSize = 8; // small vectors to get to 8 quickly
+ int samplingVectors = 8;
+ int postSamplingVectors = 4;
+ int totalValues = vectorSize * (samplingVectors + postSamplingVectors);
+
+ float[] values = new float[totalValues];
+
+ // First 8 vectors: decimal monetary values (best with e=2, f=0)
+ for (int i = 0; i < vectorSize * samplingVectors; i++) {
+ values[i] = 10.00f + (i % 100) * 0.01f; // 10.00, 10.01, 10.02, ...
+ }
+
+ // Last 4 vectors: whole numbers (best with e=0, f=0)
+ for (int i = vectorSize * samplingVectors; i < totalValues; i++) {
+ values[i] = (float) (i * 1000);
+ }
+
+ roundTripFloat(values, vectorSize);
+ }
+
+ /**
+ * Same test for doubles.
+ */
+ @Test
+ public void testDoublePresetCachingWithDistributionChange() throws Exception {
+ int vectorSize = 8;
+ int totalValues = vectorSize * 12; // 8 sampling + 4 post
+
+ double[] values = new double[totalValues];
+
+ for (int i = 0; i < vectorSize * 8; i++) {
+ values[i] = 10.00 + (i % 100) * 0.01;
+ }
+ for (int i = vectorSize * 8; i < totalValues; i++) {
+ values[i] = (double) (i * 1000);
+ }
+
+ roundTripDouble(values, vectorSize);
+ }
+
+ // ========== Multi-Vector Hand-Crafted Binary ==========
+
+ /**
+ * Hand-craft a 2-vector page to verify offset array navigation works.
+ * 10 values with vectorSize=8: vector 0 has values 1-8, vector 1 has 9-10.
+ */
+ @Test
+ public void testMultiVectorHandCrafted() throws Exception {
+ int vectorSize = 8;
+ int numElements = 10;
+ int numVectors = 2;
+
+ // Vector 0: {1,2,3,4,5,6,7,8} → e=0,f=0, encoded={1..8}, FOR min=1, deltas={0..7}
+ // maxDelta=7, bitWidth=3. Packed: 8 values at 3 bits = 3 bytes exactly.
+ int v0BitWidth = 3;
+ BytePacker packer3 = Packer.LITTLE_ENDIAN.newBytePacker(3);
+ int[] v0Deltas = {0, 1, 2, 3, 4, 5, 6, 7};
+ byte[] v0Packed = new byte[3];
+ packer3.pack8Values(v0Deltas, 0, v0Packed, 0);
+ int v0PackedSize = 3; // (8*3+7)/8 = 3
+
+ // Vector 1: {9, 10} → e=0,f=0, encoded={9,10}, FOR min=9, deltas={0,1}
+ // maxDelta=1, bitWidth=1. Packed: ceil(2*1/8) = 1 byte.
+ int v1BitWidth = 1;
+ BytePacker packer1 = Packer.LITTLE_ENDIAN.newBytePacker(1);
+ int[] v1Deltas = {0, 1, 0, 0, 0, 0, 0, 0};
+ byte[] v1Packed = new byte[1];
+ packer1.pack8Values(v1Deltas, 0, v1Packed, 0);
+ int v1PackedSize = 1; // (2*1+7)/8 = 1
+
+ int v0DataSize = 4 + 5 + v0PackedSize; // 12
+ int v1DataSize = 4 + 5 + v1PackedSize; // 10
+ int offsetArraySize = numVectors * 4; // 8
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + v0DataSize + v1DataSize)
+ .order(ByteOrder.LITTLE_ENDIAN);
+
+ // Header (7 bytes)
+ page.put((byte) 0); // compression_mode
+ page.put((byte) 0); // integer_encoding
+ page.put((byte) 3); // log2(8)
+ page.putInt(numElements);
+
+ // Offset array
+ page.putInt(offsetArraySize); // vector 0 starts at 8
+ page.putInt(offsetArraySize + v0DataSize); // vector 1 starts at 8+12=20
+
+ // Vector 0
+ page.put((byte) 0); // exponent
+ page.put((byte) 0); // factor
+ page.putShort((short) 0); // exceptions
+ page.putInt(1); // FOR = 1
+ page.put((byte) v0BitWidth);
+ page.put(v0Packed, 0, v0PackedSize);
+
+ // Vector 1
+ page.put((byte) 0); // exponent
+ page.put((byte) 0); // factor
+ page.putShort((short) 0); // exceptions
+ page.putInt(9); // FOR = 9
+ page.put((byte) v1BitWidth);
+ page.put(v1Packed, 0, v1PackedSize);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ for (int i = 0; i < numElements; i++) {
+ float expected = (float) (i + 1);
+ float actual = reader.readFloat();
+ assertEquals(
+ "Value mismatch at index " + i,
+ Float.floatToRawIntBits(expected),
+ Float.floatToRawIntBits(actual));
+ }
+ }
+
+ // ========== Skip Over Entire Vector Without Decoding ==========
+
+ /**
+ * Verify that skipping an entire vector and reading the next one works.
+ * Uses hand-crafted bytes to ensure independence from writer.
+ */
+ @Test
+ public void testSkipEntireVectorHandCrafted() throws Exception {
+ // Reuse the multi-vector setup: {1..8} in vector 0, {9, 10} in vector 1
+ int vectorSize = 8;
+ int numElements = 10;
+ int numVectors = 2;
+
+ BytePacker packer3 = Packer.LITTLE_ENDIAN.newBytePacker(3);
+ int[] v0Deltas = {0, 1, 2, 3, 4, 5, 6, 7};
+ byte[] v0Packed = new byte[3];
+ packer3.pack8Values(v0Deltas, 0, v0Packed, 0);
+
+ BytePacker packer1 = Packer.LITTLE_ENDIAN.newBytePacker(1);
+ int[] v1Deltas = {0, 1, 0, 0, 0, 0, 0, 0};
+ byte[] v1Packed = new byte[1];
+ packer1.pack8Values(v1Deltas, 0, v1Packed, 0);
+
+ int v0DataSize = 4 + 5 + 3;
+ int v1DataSize = 4 + 5 + 1;
+ int offsetArraySize = numVectors * 4;
+
+ ByteBuffer page =
+ ByteBuffer.allocate(7 + offsetArraySize + v0DataSize + v1DataSize)
+ .order(ByteOrder.LITTLE_ENDIAN);
+
+ page.put((byte) 0).put((byte) 0).put((byte) 3);
+ page.putInt(numElements);
+ page.putInt(offsetArraySize);
+ page.putInt(offsetArraySize + v0DataSize);
+
+ // Vector 0
+ page.put((byte) 0).put((byte) 0).putShort((short) 0);
+ page.putInt(1).put((byte) 3);
+ page.put(v0Packed, 0, 3);
+
+ // Vector 1
+ page.put((byte) 0).put((byte) 0).putShort((short) 0);
+ page.putInt(9).put((byte) 1);
+ page.put(v1Packed, 0, 1);
+
+ page.flip();
+
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(numElements, ByteBufferInputStream.wrap(page));
+
+ // Skip entire vector 0 (8 values)
+ reader.skip(8);
+
+ // Read from vector 1
+ assertEquals(Float.floatToRawIntBits(9.0f), Float.floatToRawIntBits(reader.readFloat()));
+ assertEquals(Float.floatToRawIntBits(10.0f), Float.floatToRawIntBits(reader.readFloat()));
+ }
+
+ // ========== Vector Size Validation ==========
+
+ /**
+ * Verify that vectorSize=65536 is rejected because num_exceptions (uint16)
+ * cannot represent 65536, which would cause silent data corruption if all
+ * values in a vector are exceptions.
+ */
+ @Test(expected = IllegalArgumentException.class)
+ public void testVectorSize65536Rejected() throws Exception {
+ new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 65536);
+ }
+
+ /**
+ * Verify that vectorSize=32768 (max allowed) works correctly.
+ */
+ @Test
+ public void testVectorSize32768Works() throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(256, 256, new DirectByteBufferAllocator(), 32768);
+ // Write a small number of values (partial vector)
+ for (int i = 0; i < 10; i++) {
+ writer.writeFloat(i * 1.0f);
+ }
+
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(10, ByteBufferInputStream.wrap(input.toByteBuffer()));
+
+ for (int i = 0; i < 10; i++) {
+ assertEquals(
+ Float.floatToRawIntBits(i * 1.0f),
+ Float.floatToRawIntBits(reader.readFloat()));
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Factor > 0 Tests ==========
+
+ /**
+ * Data where the best (e,f) combo has factor > 0.
+ * Values like 1234.56 have digits on both sides of the decimal point.
+ * With e=4,f=2: 1234.56 * 10000 / 100 = 123456 — exact integer.
+ * This exercises the division path in the encode/decode formula.
+ */
+ @Test
+ public void testDoubleFactorGreaterThanZero() throws Exception {
+ // These values are designed so the best combo has f > 0
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 1000.0 + i * 0.01; // 1000.00, 1000.01, ...
+ }
+
+ // Verify the best params actually have f > 0 or f == 0 with low exceptions
+ AlpEncoderDecoder.EncodingParams params =
+ AlpEncoderDecoder.findBestDoubleParams(values, 0, values.length);
+ // Regardless of the chosen f, the roundtrip must be lossless
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatFactorGreaterThanZero() throws Exception {
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = 1000.0f + i * 0.01f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== Overflow Boundary Tests ==========
+
+ /**
+ * Values near the encoding limits that test the overflow check.
+ * The old code used Long.MAX_VALUE (which rounds to 2^63 as double),
+ * the fix uses ENCODING_UPPER_LIMIT = 9223372036854774784.0.
+ */
+ @Test
+ public void testDoubleNearOverflowBoundary() throws Exception {
+ double[] values = {
+ 9.2e18, // near Long.MAX_VALUE
+ -9.2e18, // near Long.MIN_VALUE
+ 1.7e308, // near Double.MAX_VALUE
+ -1.7e308, // near -Double.MAX_VALUE
+ 4.9e-324, // Double.MIN_VALUE (subnormal)
+ 1.0e18, // large but in range
+ -1.0e18, // large negative but in range
+ 0.0, // normal
+ 1.23, // normal
+ };
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatNearOverflowBoundary() throws Exception {
+ float[] values = {
+ 2.1e9f, // near Integer.MAX_VALUE
+ -2.1e9f, // near Integer.MIN_VALUE
+ 3.4e38f, // near Float.MAX_VALUE
+ -3.4e38f, // near -Float.MAX_VALUE
+ 1.4e-45f, // Float.MIN_VALUE (subnormal)
+ 1.0e9f, // large but in range
+ 0.0f, // normal
+ 1.23f, // normal
+ };
+ roundTripFloat(values);
+ }
+
+ // ========== Large Scale Stress Test ==========
+
+ @Test
+ public void testDoubleLargeScale100K() throws Exception {
+ Random rand = new Random(12345);
+ double[] values = new double[100_000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextDouble() * 10000.0 - 5000.0;
+ }
+ roundTripDouble(values);
+ }
+
+ @Test
+ public void testFloatLargeScale100K() throws Exception {
+ Random rand = new Random(12345);
+ float[] values = new float[100_000];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = rand.nextFloat() * 10000.0f - 5000.0f;
+ }
+ roundTripFloat(values);
+ }
+
+ // ========== All Exceptions in Large Vector ==========
+
+ /**
+ * A full vector where every value is an exception (NaN/Inf/-0.0).
+ * This exercises the case where placeholder search finds no valid value.
+ */
+ @Test
+ public void testFloatEntireVectorAllExceptions() throws Exception {
+ float[] values = new float[DEFAULT_VECTOR_SIZE + 5]; // partial second vector
+ for (int i = 0; i < values.length; i++) {
+ switch (i % 4) {
+ case 0: values[i] = Float.NaN; break;
+ case 1: values[i] = Float.POSITIVE_INFINITY; break;
+ case 2: values[i] = Float.NEGATIVE_INFINITY; break;
+ case 3: values[i] = -0.0f; break;
+ }
+ }
+ roundTripFloat(values);
+ }
+
+ @Test
+ public void testDoubleEntireVectorAllExceptions() throws Exception {
+ double[] values = new double[DEFAULT_VECTOR_SIZE + 5];
+ for (int i = 0; i < values.length; i++) {
+ switch (i % 4) {
+ case 0: values[i] = Double.NaN; break;
+ case 1: values[i] = Double.POSITIVE_INFINITY; break;
+ case 2: values[i] = Double.NEGATIVE_INFINITY; break;
+ case 3: values[i] = -0.0; break;
+ }
+ }
+ roundTripDouble(values);
+ }
+
Review Comment:
Added testFloatPartialNullPage / testDoublePartialNullPage for exactly this: the writer sees 700 values but the reader is initialized with a page count of 1000 (so 300 null rows, num_elements < valueCount), and it serves the 700 encoded values back correctly. The all-null case (num_elements 0) is covered by testFloat/DoubleAllNullPage as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4019015378
##########
parquet-format-structures/src/main/thrift/parquet.thrift:
##########
@@ -636,6 +636,21 @@ enum Encoding {
Support for INT32, INT64 and FIXED_LEN_BYTE_ARRAY added in 2.11.
*/
BYTE_STREAM_SPLIT = 9;
+
Review Comment:
Done as #3791, which merged yesterday, and I've merged master into this branch. `parquet.thrift` here is now byte-identical to upstream 2.14.0, so the local `ALP = 10` block and the "LOCAL ADDITION" notice are gone and the entry comes straight from parquet-format.
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
Agreed, and that's now done. The one wrinkle was that #3791 had to add a temporary ALP skip back into `testEnumEquivalence` (the bump fails on master otherwise), and merging master quietly brought it back here. It's removed again in 7b48dd189, so ALP round trips in both directions like every other encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-5559330579
@wgtmac thanks again for the review, it was a genuinely useful one. I've gone through all of your comments and replied to each thread individually. A summary of where things landed:
The three reader issues are fixed in 338fb99c6. You were right that numElements could be forged up to Integer.MAX_VALUE, and while fixing it I found the allocation was reachable rather than theoretical, since MultiBufferInputStream.slice allocates before its EOF check. The vector count is bounded against the bytes actually present now, the offset array is validated up front, each vector's reads are confined by the following offset, and skip no longer overflows.
The sampling problem you spotted was the most valuable comment in the review. Your arithmetic was exactly right: reset() ran at every page boundary and cleared the sample state, so the threshold was never reached and the preset cache was never built. Fixed in eb0e5941d, with a regression test that fails without it.
The two test comments were both fair. The allocation test was inflating valuesCount rather than forging num_elements so it never reached the path it claimed to, and catchAny was swallowing Errors so it could not really fail. Both fixed in 338fb99c6.
Beyond that: ALP is configured through a single AlpConfig now with withAlp (99ae547c5), AlpConstants is package private with the constants moved to whichever class actually uses them, AlpEncoderDecoder is renamed to AlpCodec, the wildcard imports are gone and the javadoc box is fixed (78c1d182a), ALP is declared directly in parquet.thrift with the perl patch deleted (229e0d99b), and the double encoding is gone (8e743c60a).
There are three places where I would like your opinion rather than assuming I picked right:
- To make withAlp work without breaking existing callers I added getDefaultValue and getValue to ColumnProperty.Builder. It is package private so not public API, but it is a shared file rather than something ALP specific.
- dev/update-parquet-thrift.sh overwrites parquet.thrift from upstream and will silently drop the ALP entry. I put a marked notice in the enum so the loss at least shows up in a diff, but a guard in the script might be better.
- On the nested writer classes, I followed the ByteStreamSplit layout rather than choosing it deliberately. Happy to split them if you would rather ALP be internally consistent.
The one thing I cannot fix on my own is the interop test. The fixtures are proposed in apache/parquet-testing#100, which is still open, so the test skips in CI today. If you have any pull there it would help, since cross language read back matters for the vote more than for that test alone.
parquet-column is at 856 tests with no failures. Whenever you have time for another look, I would appreciate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4078572049
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpExceptionCountTest.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Measures exception counts and compression ratios using the real Spotify and Arade datasets from
+ * the parquet-testing repository (Pratik's alpFloatingPointDataset branch).
+ *
+ * <p>These are the same datasets used in the meeting discussion to compare Java vs C++ ALP
+ * exception counts. Run from the project root with the alp-test-data/ directory present:
+ *
+ * <pre>
+ * ./mvnw test -pl parquet-column -Dtest=AlpExceptionCountTest
+ * </pre>
+ *
+ * <p>To use a different data directory:
+ *
+ * <pre>
+ * ./mvnw test -pl parquet-column -Dtest=AlpExceptionCountTest \
+ * -DALP_TEST_DATA_DIR=/path/to/alp-test-data
+ * </pre>
+ */
+public class AlpExceptionCountTest {
Review Comment:
Should we remove this file as well? I think it is useful to locally verify the ALP PoC but now it does not bring too much value since it cannot run in CI and others are not easy to run it as well.
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,775 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.values.alp.AlpConfig;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility tests for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads the ALP conformance file from parquet-testing and checks that every ALP column decodes
+ * bit-identically to its PLAIN counterpart, and round-trips Java-written ALP files.
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String ALP_EXTENDED_FILE = "alp_extended.zstd.parquet";
+ private static final String CHANGESET = "09f3cdb";
+ private final InterOpTester interop = new InterOpTester();
+
+ private static final int EXTENDED_ROWS = 9032;
+ private static final int EXTENDED_ROW_GROUPS = 5;
+ private static final int EXTENDED_NULLS = 8;
+
+ /** ALP column -> PLAIN column holding the same values. */
+ private static final Map<String, String> ALP_TO_PLAIN = new LinkedHashMap<>();
+
+ static {
+ ALP_TO_PLAIN.put("float_alp_32", "float_plain");
+ ALP_TO_PLAIN.put("float_alp_1024", "float_plain");
+ ALP_TO_PLAIN.put("float_alp_4096", "float_plain");
+ ALP_TO_PLAIN.put("double_alp_32", "double_plain");
+ ALP_TO_PLAIN.put("double_alp_1024", "double_plain");
+ ALP_TO_PLAIN.put("double_alp_4096", "double_plain");
+ }
+
+ private java.nio.file.Path getExtendedFile() throws IOException {
+ return Paths.get(interop.GetInterOpFile(ALP_EXTENDED_FILE, CHANGESET).toString());
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ private static boolean isNull(Group row, String field) {
+ return row.getFieldRepetitionCount(field) == 0;
+ }
+
+ /** Raw IEEE 754 bits of a float or double field, widened to a long for exact comparison. */
+ private static long rawBits(Group row, String field, boolean isFloat) {
+ return isFloat
+ ? Float.floatToRawIntBits(row.getFloat(field, 0)) & 0xFFFFFFFFL
+ : Double.doubleToRawLongBits(row.getDouble(field, 0));
+ }
+
+ @Test
+ public void testAlpColumnsMatchPlainColumns() throws IOException {
+ List<Group> rows = readAllRows(getExtendedFile());
+ assertThat(rows.size()).as("row count").isEqualTo(EXTENDED_ROWS);
+
+ for (Map.Entry<String, String> entry : ALP_TO_PLAIN.entrySet()) {
+ String alpCol = entry.getKey();
+ String plainCol = entry.getValue();
+ boolean isFloat = alpCol.startsWith("float");
+ int nulls = 0;
+ for (int i = 0; i < rows.size(); i++) {
+ Group row = rows.get(i);
+ boolean alpNull = isNull(row, alpCol);
+ assertThat(alpNull).as("nullness of " + alpCol + " at row " + i).isEqualTo(isNull(row, plainCol));
+ if (alpNull) {
+ nulls++;
+ continue;
+ }
+ long alpBits = rawBits(row, alpCol, isFloat);
+ long plainBits = rawBits(row, plainCol, isFloat);
+ assertThat(alpBits)
+ .as(String.format(
+ "%s row %d: bits 0x%x, %s bits 0x%x", alpCol, i, alpBits, plainCol, plainBits))
+ .isEqualTo(plainBits);
+ }
+ assertThat(nulls).as(alpCol + " nulls").isEqualTo(EXTENDED_NULLS);
+ }
+ }
+
+ @Test
+ public void testAlpColumnsAreAlpEncoded() throws IOException {
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(getExtendedFile()))) {
+ ParquetMetadata footer = reader.getFooter();
+ assertThat(footer.getBlocks().size()).as("row groups").isEqualTo(EXTENDED_ROW_GROUPS);
+ int alpChunks = 0;
+ for (org.apache.parquet.hadoop.metadata.BlockMetaData block : footer.getBlocks()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData chunk : block.getColumns()) {
+ String path = chunk.getPath().toDotString();
+ boolean hasAlp = chunk.getEncodings().contains(Encoding.ALP);
+ assertThat(hasAlp)
+ .as("ALP on " + path + ": " + chunk.getEncodings())
+ .isEqualTo(ALP_TO_PLAIN.containsKey(path));
+ if (hasAlp) {
+ alpChunks++;
+ }
+ }
+ }
+ assertThat(alpChunks).as("ALP column chunks").isEqualTo(ALP_TO_PLAIN.size() * EXTENDED_ROW_GROUPS);
+ }
+ }
+
+ /**
+ * Checks exact bit patterns at known rows, so a fault that corrupted the ALP and PLAIN paths the
+ * same way would still be caught. NaN payloads must survive, since exceptions are stored verbatim.
+ */
+ @Test
+ public void testSpecialValueBitPatterns() throws IOException {
+ List<Group> rows = readAllRows(getExtendedFile());
+ assertBits(rows, 1024, 0x7FF8000000000000L, 0x7FC00000L, "canonical NaN");
+ assertBits(rows, 1500, 0x7FF800DEADBEEF00L, 0x7FC0DEADL, "NaN with payload");
+ assertBits(rows, 2047, 0xFFF8000000000001L, 0xFFC00001L, "negative NaN with payload");
+ assertBits(rows, 2000, 0x7FF0000000000000L, 0x7F800000L, "+Inf");
+ assertBits(rows, 2001, 0xFFF0000000000000L, 0xFF800000L, "-Inf");
+ assertBits(rows, 2002, 0x8000000000000000L, 0x80000000L, "-0.0");
+ assertBits(rows, 2003, 0x1L, 0x1L, "smallest subnormal");
+ assertBits(rows, 2500, 0x400921FB54442D18L, 0x40490FDBL, "pi");
+ assertBits(rows, 7168, 0x401F147AE147AE14L, 0x40F8A3D7L, "constant 7.77");
+ assertBits(rows, 9000, 0xC3DBC16D674EC800L, 0xDEDE0B6BL, "-8e18");
+ assertBits(rows, 9001, 0x43DBC16D674EC800L, 0x5EDE0B6BL, "8e18");
+ for (int row : new int[] {8200, 8900}) {
+ for (String col : ALP_TO_PLAIN.keySet()) {
+ assertThat(isNull(rows.get(row), col))
+ .as(col + " null at row " + row)
+ .isTrue();
+ }
+ }
+ }
+
+ private void assertBits(List<Group> rows, int row, long doubleBits, long floatBits, String what) {
+ for (String col : ALP_TO_PLAIN.keySet()) {
+ boolean isFloat = col.startsWith("float");
+ assertThat(rawBits(rows.get(row), col, isFloat))
+ .as(what + " at row " + row + " in " + col)
+ .isEqualTo(isFloat ? floatBits : doubleBits);
+ }
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { " + "required double double_col; " + "required float float_col; " + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlp()
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch for " + version).isEqualTo(DOUBLE_VALUES.length);
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i + " for " + version)
+ .isEqualTo(DOUBLE_VALUES[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i + " for " + version)
+ .isEqualTo(FLOAT_VALUES[i]);
+ }
+ LOG.info(
+ "writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}",
+ version,
+ rows.size(),
+ outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath = newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlp(new AlpConfig(4096))
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch at vectorSize=4096").isEqualTo(rowCount);
+ for (int i = 0; i < rowCount; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i)
+ .isEqualTo(doubles[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i)
+ .isEqualTo(floats[i]);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
+ */
+ @Test
+ public void testJavaWriteAlpV2Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_2_0);
+ }
+
+ /**
+ * Writes ALP-encoded V1 and V2 parquet files from Java and verifies that pyarrow (Python Arrow)
+ * can read them back correctly. Skipped if python3/pyarrow is not available in the environment.
+ *
+ * <p>This test addresses cross-language compatibility: Java writes, another implementation reads.
+ */
+ @Test
+ public void testJavaWrittenAlpFilesReadableByPyarrow() throws IOException, InterruptedException {
Review Comment:
Do we really need this test? It looks hacky to me. I think the C++ ALP implementation has already added an interop test by reading the file from parquet-testing repo. We can remove this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099479417
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
Review Comment:
Ah, I see: this relies on the fact that `initFromPage` calls `ByteBufferInputStream#slice` and then `ByteBuffer#slice` to ensure that the position of `vectorsData` is always 0 so that `limit` and `remaining` are the same (at least initially).
This is a brittle assumption because it allows you to use `limit` and `remaining` interchangeably when they are not the same thing. The correctness of this relies on not modifying the state of the buffer and always having a buffer that meets your assumption that position is 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099713241
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_FOR_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.PACK_GROUP_SIZE;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private final long[] unpackPadBuf = new long[PACK_GROUP_SIZE];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ protected int maxExponent() {
+ return DOUBLE_MAX_EXPONENT;
+ }
+
+ @Override
+ protected String typeName() {
+ return "double";
+ }
+
+ @Override
+ protected int forInfoSize() {
+ return DOUBLE_FOR_INFO_SIZE;
+ }
+
+ @Override
+ protected int exceptionValueSize() {
+ return Double.BYTES;
+ }
+
+ @Override
+ public double readDouble() {
+ if (pageValueIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int vectorSlot = pageValueIndex % vectorSize;
Review Comment:
Prefer consistent terms, like "index" for array indexes. The first words clarify _which_ index you're talking about, like `vectorIndex`, `pageValueIndex`, or `vectorValueIndex`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3974965466
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +637,74 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Enable or disable ALP encoding for FLOAT and DOUBLE columns.
+ *
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(boolean enable) {
+ this.alpEnabled.withDefaultValue(enable);
+ return this;
+ }
+
+ /**
+ * Enable or disable ALP encoding for the specified column.
+ *
+ * @param columnPath the path of the column (dot-string)
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(String columnPath, boolean enable) {
+ this.alpEnabled.withValue(columnPath, enable);
+ return this;
+ }
+
+ /**
+ * Set the ALP vector size (number of values per encoded vector) for FLOAT and DOUBLE columns.
+ * Must be a power of 2 in the range supported by {@link AlpConstants}.
+ *
+ * @param vectorSize the vector size
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpVectorSize(int vectorSize) {
Review Comment:
Following up since this moved on a little: Russell pushed in the same direction and the convenience setters are gone, so `withAlp(AlpConfig)` is the primary API now, as you originally suggested. That also removed the `getDefaultValue` and `getValue` additions to `ColumnProperty.Builder` I asked you about, so that question resolves itself — the details are in e3b846a2b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3973381241
##########
parquet-format-structures/src/main/thrift/parquet.thrift:
##########
@@ -636,6 +636,21 @@ enum Encoding {
Support for INT32, INT64 and FIXED_LEN_BYTE_ARRAY added in 2.11.
*/
BYTE_STREAM_SPLIT = 9;
+
Review Comment:
We should do the copy over of parquet-thrift in a precursor pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3944056151
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ public double readDouble() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
Review Comment:
Sorry for the slow reply on this one. I agree with Russell's read: this runs once per vector rather than once per value, so at the default vector size of 1024 it is well off the hot path, and it has not shown up as significant in the benchmarks. The power of two property is there if we ever want it, but I would rather not add the indirection until a profile says it matters.
For what it is worth, the per value paths are where the time actually goes, and I did find real duplicated work there recently: the exception check was encoding each value and then the caller encoded it again. That is fixed in 8e743c60a.
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
Following up on this, since the situation has changed since I last replied.
#3709 inlined parquet.thrift into the repo, so in 229e0d99b I was able to declare ALP = 10 in the Encoding enum directly and delete the perl script that used to patch the generated code. As a side effect the enum round trip now works, and I confirmed testEnumEquivalence passes with the skip removed.
I have left the skip in place for the moment though, because the condition you were actually pointing at has not really been met. ALP is in our inlined copy of the thrift file, not in released parquet-format, so removing the guard would make this test depend on a local addition. The tradeoff is that with the guard gone, anything that drops the ALP entry (dev/update-parquet-thrift.sh overwrites the file from upstream) turns into a loud test failure instead of a silent skip, which is arguably what we want.
Happy to go either way. Let me know if you would rather I take the guard out now, and otherwise it comes out when ALP is accepted upstream, as originally planned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3508951012
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
Review Comment:
Why is this unsigned here? Won't this overstimate the range when min is negative and max is not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-6305801056
@rdblue I pushed the rest of the changes for your comments. You hadn't gotten to the writer, codec and tests yet, so I applied the same feedback there too: the writer now mirrors the reader's structure (a shared base, the spec's order, static helpers), the codec got the same cleanup, and there's a new test that pins the exact bytes the writer produces. I checked byte for byte that the writer's output didn't change. I also replaced the ALP benchmark with encoding-level ones that compare against PLAIN on the same data.
Thanks to whoever approved CI — it's green on the latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099443007
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
Review Comment:
I think this is only needed if `vectorOffsets` is reused. Right now, this is always used to allocate a new `vectorOffsets` so it is equal to `vectorOffsets.length`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098784941
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java:
##########
@@ -147,7 +148,13 @@ private ValuesWriter getInt96ValuesWriter(ColumnDescriptor path) {
private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) {
final ValuesWriter fallbackWriter;
- if (this.parquetProperties.isByteStreamSplitEnabled(path)) {
+ if (this.parquetProperties.isAlpEnabled(path)) {
Review Comment:
+1 for adding this in both factories. I think we are releasing this with the ability for writers to opt in, so it should be available in both and we will control where it is enabled by default in `ParquetProperties`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098898763
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
Review Comment:
The term "interleaved" doesn't appear in the ALP spec and the diagram doesn't explain what it is. The diagram just indicates that each vector is interleaved. Can you either clarify or remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099313761
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
Review Comment:
It would be nice to reuse these allocations since we don't expect `vectorSize` to change between pages in the same column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099283527
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
Review Comment:
Minor: I don't see a need for `numVectorsLong` and would move the cast:
```java
this.numVectors = Integer.toIntExact(((long) numElements + vectorSize - 1) / vectorSize);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4103130277
##########
parquet-column/src/main/java/org/apache/parquet/column/Encoding.java:
##########
@@ -147,6 +149,26 @@ public ValuesReader getValuesReader(ColumnDescriptor descriptor, ValuesType valu
}
},
+ /**
+ * ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE types.
+ * Works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference (FOR) encoding and bit-packing.
Review Comment:
May I suggest https://parquet.apache.org/blog/2026/09/22/alp-adaptive-lossless-floating-point-encoding-in-apache-parquet/ or https://parquet.apache.org/docs/file-format/data-pages/alpencoding/
##########
parquet-column/src/main/java/org/apache/parquet/column/Encoding.java:
##########
@@ -147,6 +149,26 @@ public ValuesReader getValuesReader(ColumnDescriptor descriptor, ValuesType valu
}
},
+ /**
+ * ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE types.
+ * Works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference (FOR) encoding and bit-packing.
Review Comment:
May I suggest links to https://parquet.apache.org/blog/2026/09/22/alp-adaptive-lossless-floating-point-encoding-in-apache-parquet/ or https://parquet.apache.org/docs/file-format/data-pages/alpencoding/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "xiaoxuandev (via GitHub)" <gi...@apache.org>.
xiaoxuandev commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4289000466
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpCodec.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.FLOAT_MAX_EXPONENT;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpCodec {
+
+ // Magic numbers for the fast-rounding trick (see ALP paper, Section 3.2)
+ static final float MAGIC_FLOAT = 12_582_912.0f; // 2^22 + 2^23
+ static final double MAGIC_DOUBLE = 6_755_399_441_055_744.0; // 2^51 + 2^52
+
+ // POWERS_OF_TEN: positive powers used for scaling up during encode/decode.
+ // Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ // Decode: encoded * POW10[f] * POW10_NEGATIVE[e]
+ static final float[] FLOAT_POW10 = {1e0f, 1e1f, 1e2f, 1e3f, 1e4f, 1e5f, 1e6f, 1e7f, 1e8f, 1e9f, 1e10f};
+
+ static final double[] DOUBLE_POW10 = {
+ 1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18
+ };
+
+ // NEGATIVE_POWERS_OF_TEN: reciprocals used for scaling down (multiply-by-reciprocal).
+ // Multiplying by a precomputed reciprocal instead of dividing keeps the rounding bit-for-bit
+ // identical to the ALP reference implementation, so encoded values interop across implementations.
+ static final float[] FLOAT_POW10_NEGATIVE = {
+ 1e0f, 1e-1f, 1e-2f, 1e-3f, 1e-4f, 1e-5f, 1e-6f, 1e-7f, 1e-8f, 1e-9f, 1e-10f
+ };
+
+ static final double[] DOUBLE_POW10_NEGATIVE = {
+ 1e0, 1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7, 1e-8, 1e-9, 1e-10, 1e-11, 1e-12, 1e-13, 1e-14, 1e-15, 1e-16,
+ 1e-17, 1e-18
+ };
+
+ static final int FLOAT_NEGATIVE_ZERO_BITS = 0x80000000;
+ static final long DOUBLE_NEGATIVE_ZERO_BITS = 0x8000000000000000L;
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpCodec() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /**
+ * Encodes {@code value} and reports whether it is an exception, doing the scaling and rounding
+ * once and leaving the outcome in {@code out}.
+ *
+ * <p>Callers that need both the flag and the encoded value should use this rather than
+ * {@link #isFloatException} followed by {@link #encodeFloat}: the exception check has to encode
+ * the value to test the round trip, so the pair encodes twice. The scaled intermediate is reused
+ * here too, so the two multiplies are also only done once. The expression and its order match
+ * {@link #encodeFloat} exactly, which matters for cross-implementation bit compatibility.
+ */
+ static void tryEncodeFloat(float value, int exponent, int factor, EncodeResult out) {
+ if (isIntrinsicFloatException(value)) {
+ out.isException = true;
+ return;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ out.isException = true;
+ return;
+ }
+ int encoded = fastRoundFloat(scaled);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ if (Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded)) {
+ out.isException = true;
+ return;
+ }
+ out.encoded = encoded;
+ out.isException = false;
+ }
+
+ /**
+ * Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure.
+ * Prefer {@link #tryEncodeFloat} when the encoded value is wanted as well.
+ */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ EncodeResult result = new EncodeResult();
+ tryEncodeFloat(value, exponent, factor, result);
+ return result.isException;
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ /**
+ * Encodes {@code value} and reports whether it is an exception, doing the scaling and rounding
+ * once and leaving the outcome in {@code out}. See {@link #tryEncodeFloat} for why this exists.
+ */
+ static void tryEncodeDouble(double value, int exponent, int factor, EncodeResult out) {
+ if (isIntrinsicDoubleException(value)) {
+ out.isException = true;
+ return;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ out.isException = true;
+ return;
+ }
+ long encoded = fastRoundDouble(scaled);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ if (Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded)) {
+ out.isException = true;
+ return;
+ }
+ out.encoded = encoded;
+ out.isException = false;
+ }
+
+ /**
+ * Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure.
+ * Prefer {@link #tryEncodeDouble} when the encoded value is wanted as well.
+ */
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ EncodeResult result = new EncodeResult();
+ tryEncodeDouble(value, exponent, factor, result);
+ return result.isException;
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /**
+ * Outcome of a single {@link #tryEncodeFloat} or {@link #tryEncodeDouble} call. Mutable and
+ * meant to be reused by the caller so the per-value hot path allocates nothing; an Optional or a
+ * freshly allocated result here would box on every value. Not thread safe, so each writer keeps
+ * its own.
+ */
+ static final class EncodeResult {
+ /** The encoded value, meaningful only when {@link #isException} is false. */
+ long encoded;
+
+ boolean isException;
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ // Index positions within an (exponent, factor) pair array.
+ private static final int E = 0;
+ private static final int F = 1;
+
+ // All valid (exponent, factor) pairs for the full search, precomputed in nested-loop order
+ // (e ascending, then f ascending) so the tie-break and early-exit behave exactly like an inline
+ // double loop. Reused across every vector to avoid per-call allocation.
+ private static final int[][] ALL_VALID_FLOAT_PAIRS = buildAllPairs(FLOAT_MAX_EXPONENT);
+ private static final int[][] ALL_VALID_DOUBLE_PAIRS = buildAllPairs(DOUBLE_MAX_EXPONENT);
+
+ private static int[][] buildAllPairs(int maxExponent) {
+ int count = (maxExponent + 1) * (maxExponent + 2) / 2;
+ int[][] pairs = new int[count][];
+ int idx = 0;
+ for (int e = 0; e <= maxExponent; e++) {
+ for (int f = 0; f <= e; f++) {
+ pairs[idx++] = new int[] {e, f};
+ }
+ }
+ return pairs;
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ return pickBestFloat(values, offset, length, ALL_VALID_FLOAT_PAIRS);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ return pickBestFloat(values, offset, length, presets);
+ }
+
+ /**
+ * Scores each (exponent, factor) pair by estimated compressed size and returns the best.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the signed range (max - min) of
+ * non-exception encoded values after frame-of-reference subtraction, matching the writer's FOR
+ * packing. This produces better compression ratios than minimizing exception count alone. Ties in
+ * size are broken toward the higher exponent, then the higher factor. The first pair (in {@code
+ * pairs} order) that encodes every value into a single FOR value with zero exceptions wins
+ * outright. When no pair yields any non-exception values, {@code pairs[0]} is returned.
+ */
+ private static EncodingParams pickBestFloat(float[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ // One reusable result for the whole search: every value below is encoded exactly once.
+ EncodeResult result = new EncodeResult();
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ tryEncodeFloat(values[offset + i], e, f, result);
+ if (result.isException) {
+ exceptions++;
+ } else {
+ int encoded = (int) result.encoded;
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ // Signed subtraction gives the true FOR span; getWidthFromMaxLong yields the correct
+ // unsigned bit width, and an overflowing large range is penalized with 64 bits. This matches
+ // the writer's FOR packing.
+ long delta = (nonExceptions < 2) ? 0 : ((long) maxEncoded - (long) minEncoded);
+ int bitsPerValue = BytesUtils.getWidthFromMaxLong(delta);
+ long estimatedSize = (long) length * bitsPerValue + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestDoubleParams(double[] values, int offset, int length) {
+ return pickBestDouble(values, offset, length, ALL_VALID_DOUBLE_PAIRS);
+ }
+
+ /** Same as findBestDoubleParams but only tries the cached preset combos. */
+ static EncodingParams findBestDoubleParamsWithPresets(double[] values, int offset, int length, int[][] presets) {
+ return pickBestDouble(values, offset, length, presets);
+ }
+
+ /** Double counterpart to {@link #pickBestFloat}; see it for the scoring and tie-break rules. */
+ private static EncodingParams pickBestDouble(double[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ // One reusable result for the whole search: every value below is encoded exactly once.
+ EncodeResult result = new EncodeResult();
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ long minEncoded = Long.MAX_VALUE;
+ long maxEncoded = Long.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ tryEncodeDouble(values[offset + i], e, f, result);
Review Comment:
Not blocking, and fine as a follow-up given how much this PR has already been through.
I was curious what the parameter search costs, so I timed `DoubleAlpValuesWriter` on its own (1M doubles, same data shape as the benchmark, 20480-value pages): about 70 ns/value vs ~7 ns/value for `PlainValuesWriter`. Most of that looks like this loop, since each candidate pair encodes the full vector. The spec's Parameter Selection section describes sampling as an option (256 values/vector) and the reference implementation uses 32; trying a strided sample here brought it to roughly 24 ns and 8 ns, with byte-identical output on the decimal data, and the result still decodes bit-exactly through the reader.
One thing I noticed while trying it: on random doubles the sampled version produced ~27% larger output, because a sample with a single encodable value scores bitWidth 0 while the full vector then needs ~60 bits. So if you go this way, the mostly-exceptions case probably wants special handling. Sharing in case it's useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3973354419
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -415,7 +481,8 @@ public String toString() {
+ "Page row count limit to " + getPageRowCountLimit() + '\n'
+ "Writing page checksums is: " + (getPageWriteChecksumEnabled() ? "on" : "off") + '\n'
+ "Statistics enabled: " + statisticsEnabled + '\n'
- + "Size statistics enabled: " + sizeStatisticsEnabled;
+ + "Size statistics enabled: " + sizeStatisticsEnabled + '\n'
+ + "ALP: " + alp;
Review Comment:
nit: A little odd here since it will say "ALP: null" when unset probably worth a bit more clarity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3970313674
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +624,77 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Set the full ALP configuration for FLOAT and DOUBLE columns.
+ *
+ * @param config the ALP configuration
+ * @return this builder for method chaining.
+ */
+ public Builder withAlp(AlpConfig config) {
Review Comment:
What do you think? Do we need per column disabling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4305395884
##########
parquet-column/src/main/java/org/apache/parquet/column/Encoding.java:
##########
@@ -147,6 +149,26 @@ public ValuesReader getValuesReader(ColumnDescriptor descriptor, ValuesType valu
}
},
+ /**
+ * ALP (Adaptive Lossless floating-Point) encoding for FLOAT and DOUBLE types.
+ * Works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference (FOR) encoding and bit-packing.
Review Comment:
Cut it down to one line plus links to the spec and the blog post @alamb suggested. Most of the other constants in `Encoding` have a short javadoc, so I kept the one line, but I can remove it if you'd prefer.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
Review Comment:
Removed both, and the same description from the writer's javadoc.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
Review Comment:
Removed it here, in the float and double readers, and in the writer.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
Review Comment:
Removed it. The javadoc on `getVectorDataPosition` now says offsets are measured from the start of the offset array.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
Review Comment:
Updated the javadoc to say it's a position in the vector data, not an offset as the spec defines one.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpCodec.java:
##########
@@ -0,0 +1,394 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.FLOAT_MAX_EXPONENT;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpCodec {
+
+ // Magic numbers for the fast-rounding trick (see ALP paper, Section 3.2)
+ static final float MAGIC_FLOAT = 12_582_912.0f; // 2^22 + 2^23
+ static final double MAGIC_DOUBLE = 6_755_399_441_055_744.0; // 2^51 + 2^52
+
+ // POWERS_OF_TEN: positive powers used for scaling up during encode/decode.
+ // Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ // Decode: encoded * POW10[f] * POW10_NEGATIVE[e]
+ static final float[] FLOAT_POW10 = {1e0f, 1e1f, 1e2f, 1e3f, 1e4f, 1e5f, 1e6f, 1e7f, 1e8f, 1e9f, 1e10f};
+
+ static final double[] DOUBLE_POW10 = {
+ 1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10, 1e11, 1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18
+ };
+
+ // NEGATIVE_POWERS_OF_TEN: reciprocals used for scaling down (multiply-by-reciprocal).
+ // Multiplying by a precomputed reciprocal instead of dividing keeps the rounding bit-for-bit
+ // identical to the ALP reference implementation, so encoded values interop across implementations.
+ static final float[] FLOAT_POW10_NEGATIVE = {
+ 1e0f, 1e-1f, 1e-2f, 1e-3f, 1e-4f, 1e-5f, 1e-6f, 1e-7f, 1e-8f, 1e-9f, 1e-10f
+ };
+
+ static final double[] DOUBLE_POW10_NEGATIVE = {
+ 1e0, 1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7, 1e-8, 1e-9, 1e-10, 1e-11, 1e-12, 1e-13, 1e-14, 1e-15, 1e-16,
+ 1e-17, 1e-18
+ };
+
+ static final int FLOAT_NEGATIVE_ZERO_BITS = 0x80000000;
+ static final long DOUBLE_NEGATIVE_ZERO_BITS = 0x8000000000000000L;
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpCodec() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /**
+ * Encodes {@code value} and reports whether it is an exception, doing the scaling and rounding
+ * once and leaving the outcome in {@code out}.
+ *
+ * <p>Callers that need both the flag and the encoded value should use this rather than
+ * {@link #isFloatException} followed by {@link #encodeFloat}: the exception check has to encode
+ * the value to test the round trip, so the pair encodes twice. The scaled intermediate is reused
+ * here too, so the two multiplies are also only done once. The expression and its order match
+ * {@link #encodeFloat} exactly, which matters for cross-implementation bit compatibility.
+ */
+ static void tryEncodeFloat(float value, int exponent, int factor, EncodeResult out) {
+ if (isIntrinsicFloatException(value)) {
+ out.isException = true;
+ return;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ out.isException = true;
+ return;
+ }
+ int encoded = fastRoundFloat(scaled);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ if (Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded)) {
+ out.isException = true;
+ return;
+ }
+ out.encoded = encoded;
+ out.isException = false;
+ }
+
+ /**
+ * Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure.
+ * Prefer {@link #tryEncodeFloat} when the encoded value is wanted as well.
+ */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ EncodeResult result = new EncodeResult();
+ tryEncodeFloat(value, exponent, factor, result);
+ return result.isException;
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ /**
+ * Encodes {@code value} and reports whether it is an exception, doing the scaling and rounding
+ * once and leaving the outcome in {@code out}. See {@link #tryEncodeFloat} for why this exists.
+ */
+ static void tryEncodeDouble(double value, int exponent, int factor, EncodeResult out) {
+ if (isIntrinsicDoubleException(value)) {
+ out.isException = true;
+ return;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ out.isException = true;
+ return;
+ }
+ long encoded = fastRoundDouble(scaled);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ if (Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded)) {
+ out.isException = true;
+ return;
+ }
+ out.encoded = encoded;
+ out.isException = false;
+ }
+
+ /**
+ * Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure.
+ * Prefer {@link #tryEncodeDouble} when the encoded value is wanted as well.
+ */
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ EncodeResult result = new EncodeResult();
+ tryEncodeDouble(value, exponent, factor, result);
+ return result.isException;
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /**
+ * Outcome of a single {@link #tryEncodeFloat} or {@link #tryEncodeDouble} call. Mutable and
+ * meant to be reused by the caller so the per-value hot path allocates nothing; an Optional or a
+ * freshly allocated result here would box on every value. Not thread safe, so each writer keeps
+ * its own.
+ */
+ static final class EncodeResult {
+ /** The encoded value, meaningful only when {@link #isException} is false. */
+ long encoded;
+
+ boolean isException;
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ // Index positions within an (exponent, factor) pair array.
+ private static final int E = 0;
+ private static final int F = 1;
+
+ // All valid (exponent, factor) pairs for the full search, precomputed in nested-loop order
+ // (e ascending, then f ascending) so the tie-break and early-exit behave exactly like an inline
+ // double loop. Reused across every vector to avoid per-call allocation.
+ private static final int[][] ALL_VALID_FLOAT_PAIRS = buildAllPairs(FLOAT_MAX_EXPONENT);
+ private static final int[][] ALL_VALID_DOUBLE_PAIRS = buildAllPairs(DOUBLE_MAX_EXPONENT);
+
+ private static int[][] buildAllPairs(int maxExponent) {
+ int count = (maxExponent + 1) * (maxExponent + 2) / 2;
+ int[][] pairs = new int[count][];
+ int idx = 0;
+ for (int e = 0; e <= maxExponent; e++) {
+ for (int f = 0; f <= e; f++) {
+ pairs[idx++] = new int[] {e, f};
+ }
+ }
+ return pairs;
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ return pickBestFloat(values, offset, length, ALL_VALID_FLOAT_PAIRS);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ return pickBestFloat(values, offset, length, presets);
+ }
+
+ /**
+ * Scores each (exponent, factor) pair by estimated compressed size and returns the best.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the signed range (max - min) of
+ * non-exception encoded values after frame-of-reference subtraction, matching the writer's FOR
+ * packing. This produces better compression ratios than minimizing exception count alone. Ties in
+ * size are broken toward the higher exponent, then the higher factor. The first pair (in {@code
+ * pairs} order) that encodes every value into a single FOR value with zero exceptions wins
+ * outright. When no pair yields any non-exception values, {@code pairs[0]} is returned.
+ */
+ private static EncodingParams pickBestFloat(float[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ // One reusable result for the whole search: every value below is encoded exactly once.
+ EncodeResult result = new EncodeResult();
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ tryEncodeFloat(values[offset + i], e, f, result);
+ if (result.isException) {
+ exceptions++;
+ } else {
+ int encoded = (int) result.encoded;
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ // Signed subtraction gives the true FOR span; getWidthFromMaxLong yields the correct
+ // unsigned bit width, and an overflowing large range is penalized with 64 bits. This matches
+ // the writer's FOR packing.
+ long delta = (nonExceptions < 2) ? 0 : ((long) maxEncoded - (long) minEncoded);
+ int bitsPerValue = BytesUtils.getWidthFromMaxLong(delta);
+ long estimatedSize = (long) length * bitsPerValue + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
+ || (estimatedSize == bestEstimatedSize
+ && (e > bestExponent || (e == bestExponent && f > bestFactor)))) {
+ bestEstimatedSize = estimatedSize;
+ bestExponent = e;
+ bestFactor = f;
+ bestExceptions = exceptions;
+ if (bestExceptions == 0 && bitsPerValue == 0) {
+ return new EncodingParams(bestExponent, bestFactor, 0);
+ }
+ }
+ }
+ return new EncodingParams(bestExponent, bestFactor, bestExceptions);
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestDoubleParams(double[] values, int offset, int length) {
+ return pickBestDouble(values, offset, length, ALL_VALID_DOUBLE_PAIRS);
+ }
+
+ /** Same as findBestDoubleParams but only tries the cached preset combos. */
+ static EncodingParams findBestDoubleParamsWithPresets(double[] values, int offset, int length, int[][] presets) {
+ return pickBestDouble(values, offset, length, presets);
+ }
+
+ /** Double counterpart to {@link #pickBestFloat}; see it for the scoring and tie-break rules. */
+ private static EncodingParams pickBestDouble(double[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ // One reusable result for the whole search: every value below is encoded exactly once.
+ EncodeResult result = new EncodeResult();
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ long minEncoded = Long.MAX_VALUE;
+ long maxEncoded = Long.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ tryEncodeDouble(values[offset + i], e, f, result);
Review Comment:
Thanks for digging into this, those numbers are really helpful. I agree it's the full search over every (e, f) pair. The random doubles case is a good catch: if the sample happens to contain only one encodable value it scores bit width 0, while the whole vector needs around 60 bits, so sampling needs some protection for vectors that are mostly exceptions. I'd rather not change what the writer produces this close to the vote, so I opened #3863 to follow up, using your numbers as the starting point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3567821116
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1066 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
Review Comment:
Yeah, covered now. testJavaWriteAlpV1Pages and testJavaWriteAlpV2Pages write ALP from Java and read it back, and testJavaWrittenAlpFilesReadableByPyarrow writes from Java and reads it with pyarrow (it skips cleanly until Arrow ships ALP read support).
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1066 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @Rule
+ public TemporaryFolder temp = new TemporaryFolder();
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {
+ "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
+ if (dir == null) dir = System.getenv("ALP_TEST_DATA_DIR");
+ if (dir != null && new File(dir).isDirectory()) return Paths.get(dir);
+ // Default: alp-test-data/ relative to project root (two levels up from target/test-classes)
+ java.nio.file.Path candidate =
+ Paths.get(System.getProperty("user.dir")).resolve("alp-test-data");
+ return candidate.toFile().isDirectory() ? candidate : null;
+ }
+
+ private java.nio.file.Path getSingleTestFile() {
+ String filePath = System.getProperty("ALP_TEST_FILE");
+ if (filePath == null) filePath = System.getenv("ALP_TEST_FILE");
+ if (filePath != null && new File(filePath).exists()) return Paths.get(filePath);
+ return null;
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ @Test
+ public void testReadSingleAlpFile() throws IOException {
+ java.nio.file.Path file = getSingleTestFile();
+ assumeTrue("ALP_TEST_FILE not set or file does not exist, skipping", file != null);
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected at least one row in " + file, rows.size() > 0);
+ LOG.info("testReadSingleAlpFile: read {} rows from {}", rows.size(), file.getFileName());
+ }
+
+ @Test
+ public void testReadCppDoubleFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_DOUBLE_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppDoubleFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ @Test
+ public void testReadCppFloatFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ for (String filename : CPP_FLOAT_FILES) {
+ java.nio.file.Path file = dir.resolve(filename);
+ assumeTrue("File not found: " + file, file.toFile().exists());
+ List<Group> rows = readAllRows(file);
+ assertTrue("Expected rows in " + filename, rows.size() > 0);
+ LOG.info("testReadCppFloatFiles: {} → {} rows OK", filename, rows.size());
+ }
+ }
+
+ /** Verify no NaN/Inf corruption in float/double columns across all test files. */
+ @Test
+ public void testNoCorruptionInCppFiles() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ int nanCount = 0;
+ int infCount = 0;
+ int totalValues = 0;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < pages.getRowCount(); i++) {
+ Group row = recordReader.read();
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ try {
+ if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float v = row.getFloat(f, 0);
+ totalValues++;
+ if (Float.isNaN(v)) nanCount++;
+ if (Float.isInfinite(v)) infCount++;
+ } else if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double v = row.getDouble(f, 0);
+ totalValues++;
+ if (Double.isNaN(v)) nanCount++;
+ if (Double.isInfinite(v)) infCount++;
+ }
+ } catch (Exception ignored) {
+ }
+ }
+ }
+ }
+ }
+ LOG.info(
+ "{}: {} values, {} NaN, {} Inf", filename, totalValues, nanCount, infCount);
+ assertEquals("Unexpected NaN in " + filename, 0, nanCount);
+ assertEquals("Unexpected Inf in " + filename, 0, infCount);
+ }
+ }
+
+ /** Schema inspection: log encoding types used in each column. */
+ @Test
+ public void testLogSchemaAndEncodings() throws IOException {
+ java.nio.file.Path dir = getTestDataDir();
+ assumeTrue("alp-test-data/ directory not found, skipping", dir != null);
+ String[] allFiles = {
+ "alp_spotify1.parquet",
+ "alp_arade.parquet",
+ "alp_float_spotify1.parquet",
+ "alp_float_arade.parquet"
+ };
+ for (String filename : allFiles) {
+ java.nio.file.Path file = dir.resolve(filename);
+ if (!file.toFile().exists()) continue;
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(file))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ LOG.info("=== {} ===", filename);
+ LOG.info("Schema: {}", schema);
+ if (!footer.getBlocks().isEmpty()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col :
+ footer.getBlocks().get(0).getColumns()) {
+ LOG.info(
+ " column={} encodings={} compression={}",
+ col.getPath(),
+ col.getEncodings(),
+ col.getCodec());
+ }
+ }
+ } catch (Exception e) {
+ LOG.warn("Could not read metadata for {}: {}", filename, e.getMessage());
+ }
+ }
+ // This test always passes — it's for inspection
+ assertTrue(true);
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { "
+ + "required double double_col; "
+ + "required float float_col; "
+ + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch for " + version, DOUBLE_VALUES.length, rows.size());
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i + " for " + version,
+ DOUBLE_VALUES[i],
+ rows.get(i).getDouble("double_col", 0),
+ 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i + " for " + version,
+ FLOAT_VALUES[i],
+ rows.get(i).getFloat("float_col", 0),
+ 0.0f);
+ }
+ LOG.info("writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}", version, rows.size(), outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(4096)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertEquals("Row count mismatch at vectorSize=4096", rowCount, rows.size());
+ for (int i = 0; i < rowCount; i++) {
+ assertEquals(
+ "double_col mismatch at row " + i, doubles[i], rows.get(i).getDouble("double_col", 0), 0.0);
+ assertEquals(
+ "float_col mismatch at row " + i, floats[i], rows.get(i).getFloat("float_col", 0), 0.0f);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
+ * V2 page headers include the encoding value directly; ALP = 10 is supported via the build-time
+ * patch to the generated Encoding enum in parquet-format-structures.
+ */
+ @Test
+ public void testJavaWriteAlpV2Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_2_0);
+ }
+
+ /**
+ * Writes ALP-encoded V1 and V2 parquet files from Java and verifies that pyarrow (Python Arrow)
+ * can read them back correctly. Skipped if python3/pyarrow is not available in the environment.
+ *
+ * <p>This test addresses cross-language compatibility: Java writes, another implementation reads.
+ */
+ @Test
+ public void testJavaWrittenAlpFilesReadableByPyarrow() throws IOException, InterruptedException {
+ // Check python3 is available
+ Process check = new ProcessBuilder("python3", "-c", "import pyarrow.parquet")
+ .redirectErrorStream(true)
+ .start();
+ assumeTrue("python3/pyarrow not available, skipping cross-language test", check.waitFor() == 0);
+
+ for (WriterVersion version : new WriterVersion[] {WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0}) {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ temp.newFolder().toPath().resolve("alp_java_xcompat_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlpEncoding(true)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ // Build a python snippet that reads the file and asserts row count and values
+ StringBuilder pyScript = new StringBuilder();
+ pyScript.append("import pyarrow.parquet as pq, sys\n");
+ pyScript.append("t = pq.read_table('").append(outPath.toAbsolutePath()).append("')\n");
+ pyScript.append("assert len(t) == ").append(DOUBLE_VALUES.length).append(", f'row count mismatch: {len(t)}'\n");
+ pyScript.append("dc = t.column('double_col').to_pylist()\n");
+ pyScript.append("fc = t.column('float_col').to_pylist()\n");
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ pyScript.append("assert abs(dc[").append(i).append("] - ").append(DOUBLE_VALUES[i])
+ .append(") < 1e-9, f'double mismatch at ").append(i).append(": {dc[").append(i).append("]}'\n");
+ pyScript.append("assert abs(fc[").append(i).append("] - ").append(FLOAT_VALUES[i])
+ .append(") < 1e-4, f'float mismatch at ").append(i).append(": {fc[").append(i).append("]}'\n");
+ }
+ pyScript.append("print('OK: " + version.name() + " " + DOUBLE_VALUES.length + " rows verified by pyarrow')\n");
+
+ Process proc = new ProcessBuilder("python3", "-c", pyScript.toString())
+ .redirectErrorStream(true)
+ .start();
+ String output = new String(proc.getInputStream().readAllBytes());
+ int exitCode = proc.waitFor();
+ LOG.info("pyarrow cross-compat [{}]: {}", version, output.trim());
+ // pyarrow may not yet support reading ALP (Arrow C++ PR #48345 in progress).
+ // Skip rather than fail so the test becomes a passing signal once pyarrow adds ALP support.
+ assumeTrue(
+ "pyarrow does not yet support reading ALP encoding (Arrow C++ PR #48345 pending): "
+ + output,
+ !output.contains("Unknown encoding type"));
+ assertEquals(
+ "pyarrow failed to read Java-written ALP file (" + version + "): " + output, 0, exitCode);
+ }
+ }
+
+ // TODO: Uncomment once parquet-testing PR #100 merges
+ /*
+ @Test
+ public void testReadAlpFromParquetTesting() throws IOException {
+ // InterOpTester will auto-download from parquet-testing
+ }
+ */
+
+ // ---------------------------------------------------------------------------
+ // Fixture generator: reads the C++-written ALP files from parquet-testing
+ // PR #100 and re-encodes each as Java ALP at two vector sizes (1024 and 4096).
+ // Outputs land in ALP_OUTPUT_DIR (default: ${user.dir}/alp-java-generated/)
+ // and are intended for submission back to parquet-testing PR #100 so the
+ // C++/Rust/Go readers can verify cross-language compatibility against Java
+ // writes at both vector sizes.
+ // ---------------------------------------------------------------------------
+
+ private static final String[] SOURCE_FILES = {
+ "alp_spotify1.parquet", "alp_arade.parquet", "alp_float_spotify1.parquet", "alp_float_arade.parquet"
+ };
+ private static final int[] GENERATOR_VECTOR_SIZES = {1024, 4096};
+ private static final WriterVersion[] GENERATOR_PAGE_VERSIONS = {
+ WriterVersion.PARQUET_1_0, WriterVersion.PARQUET_2_0
+ };
+
+ /**
+ * Resolves the explicit output directory from {@code ALP_OUTPUT_DIR} (system property or env var)
+ * and creates it if missing. Returns {@code null} when the variable is unset — callers should
+ * {@code assumeTrue} on the result so the test skips cleanly rather than writing stray files
+ * into the module tree (which would fail the RAT license check on subsequent {@code mvn test}
+ * runs in the same module).
+ */
+ private java.nio.file.Path getOutputDir() throws IOException {
+ String dir = System.getProperty("ALP_OUTPUT_DIR");
+ if (dir == null) dir = System.getenv("ALP_OUTPUT_DIR");
+ if (dir == null) return null;
+ java.nio.file.Path target = Paths.get(dir);
+ java.nio.file.Files.createDirectories(target);
+ return target;
+ }
+
+ /**
+ * Read all rows of the given source file and return them as a list of value-only Group rows
+ * keyed by the source's schema. Used for both the copy and the verification round-trip.
+ */
+ private List<Group> readGroupsWithSchema(java.nio.file.Path source, MessageType[] outSchema) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(source))) {
+ MessageType schema = reader.getFooter().getFileMetaData().getSchema();
+ outSchema[0] = schema;
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ /**
+ * Generates Java-written ALP fixtures from the four PR #100 source datasets across two axes:
+ * page version (V1, V2) and vectorSize (1024, 4096). Each output is verified bit-exact against
+ * its source. Produces 16 files total:
+ * {@code alp_java_{spotify1,arade,float_spotify1,float_arade}_v{1,2}_vs{1024,4096}.parquet}.
+ *
+ * <p>Page version is orthogonal to ALP encoding (the page version difference lives in the
+ * parquet protocol layer, not in the ALP payload), but covering both axes makes the fixture
+ * set fully symmetric for cross-language compatibility — every reader can verify it handles
+ * Java-written ALP regardless of how the surrounding pages are framed.
+ *
+ * <p>To run: clone https://github.com/prtkgaur/parquet-testing/tree/alpFloatingPointDataset
+ * and point ALP_TEST_DATA_DIR at its {@code data/} directory. Set ALP_OUTPUT_DIR to choose
+ * where to write the generated fixtures; if unset, this test skips.
+ */
+ @Test
+ public void generateAlpFixturesAtMultipleVectorSizes() throws IOException {
+ java.nio.file.Path sourceDir = getTestDataDir();
+ assumeTrue("ALP source dir not found, skipping fixture generator", sourceDir != null);
+
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping fixture generator", outDir != null);
+ LOG.info("Generating ALP fixtures to {}", outDir);
+
+ int expectedFiles =
+ SOURCE_FILES.length * GENERATOR_PAGE_VERSIONS.length * GENERATOR_VECTOR_SIZES.length;
+ int generated = 0;
+ for (String sourceFile : SOURCE_FILES) {
+ java.nio.file.Path source = sourceDir.resolve(sourceFile);
+ assumeTrue("Source missing: " + source, source.toFile().exists());
+
+ MessageType[] schemaHolder = new MessageType[1];
+ List<Group> sourceRows = readGroupsWithSchema(source, schemaHolder);
+ MessageType schema = schemaHolder[0];
+
+ String stem = sourceFile.replace("alp_", "").replace(".parquet", "");
+
+ for (WriterVersion pageVersion : GENERATOR_PAGE_VERSIONS) {
+ String pageTag = (pageVersion == WriterVersion.PARQUET_1_0) ? "v1" : "v2";
+ for (int vectorSize : GENERATOR_VECTOR_SIZES) {
+ java.nio.file.Path outPath =
+ outDir.resolve(
+ "alp_java_" + stem + "_" + pageTag + "_vs" + vectorSize + ".parquet");
+ java.nio.file.Files.deleteIfExists(outPath);
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(pageVersion)
+ .withAlpEncoding(true)
+ .withAlpVectorSize(vectorSize)
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (Group row : sourceRows) {
+ writer.write(row);
+ }
+ }
+
+ // Verify: read back and compare against the source row-by-row
+ List<Group> roundTrip = readGroupsWithSchema(outPath, new MessageType[1]);
+ assertEquals(
+ "Row count mismatch for " + outPath.getFileName(),
+ sourceRows.size(),
+ roundTrip.size());
+ int fieldCount = schema.getFieldCount();
+ for (int i = 0; i < sourceRows.size(); i++) {
+ for (int f = 0; f < fieldCount; f++) {
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ String fieldName = schema.getType(f).getName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ double expected = sourceRows.get(i).getDouble(f, 0);
+ double actual = roundTrip.get(i).getDouble(f, 0);
+ long eb = Double.doubleToRawLongBits(expected);
+ long ab = Double.doubleToRawLongBits(actual);
+ assertEquals(
+ "double bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ float expected = sourceRows.get(i).getFloat(f, 0);
+ float actual = roundTrip.get(i).getFloat(f, 0);
+ int eb = Float.floatToRawIntBits(expected);
+ int ab = Float.floatToRawIntBits(actual);
+ assertEquals(
+ "float bit mismatch in " + outPath.getFileName() + " row " + i + " field " + fieldName,
+ eb,
+ ab);
+ }
+ }
+ }
+ long bytes = outPath.toFile().length();
+ LOG.info(
+ " wrote {} ({} rows, {} bytes, pageVersion={}, vectorSize={})",
+ outPath.getFileName(),
+ sourceRows.size(),
+ bytes,
+ pageTag,
+ vectorSize);
+ generated++;
+ }
+ }
+ }
+ assertEquals(
+ "Expected to generate " + expectedFiles + " fixture files", expectedFiles, generated);
+ LOG.info("Generated {} ALP fixture files to {}", generated, outDir);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Reader-only validation: opens each Java-written fixture and asserts the
+ // reader path works end-to-end (metadata declares ALP, every row decodes
+ // without error). Separate from the generator's own round-trip check so it
+ // surfaces as a distinct signal — useful when the fixtures already exist on
+ // disk and you want to validate just the read path.
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void readAllFixtureFilesIndependently() throws IOException {
+ java.nio.file.Path outDir = getOutputDir();
+ assumeTrue("ALP_OUTPUT_DIR not set, skipping reader-only test", outDir != null);
+ File[] files =
+ outDir.toFile().listFiles((f, n) -> n.startsWith("alp_java_") && n.endsWith(".parquet"));
+ assumeTrue("No fixture files in " + outDir, files != null && files.length > 0);
+ java.util.Arrays.sort(files, (a, b) -> a.getName().compareTo(b.getName()));
+
+ int filesRead = 0;
+ int totalChunks = 0;
+ int alpChunks = 0;
+ long totalRows = 0;
+ for (File pf : files) {
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(pf.toPath()))) {
+ ParquetMetadata footer = reader.getFooter();
+ // Layer 1: every column chunk declares ALP
+ for (org.apache.parquet.hadoop.metadata.BlockMetaData block : footer.getBlocks()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData col : block.getColumns()) {
+ totalChunks++;
+ if (col.getEncodings().contains(Encoding.ALP)) alpChunks++;
+ }
+ }
+ // Layer 2: every row actually decodes (no exceptions thrown, count matches footer)
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ long rowsDecoded = 0;
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader =
+ columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ Group row = recordReader.read();
+ // Touch every present field so any decode-time exception surfaces.
+ // Optional/null fields have repetitionCount=0 — skip those rather than throw.
+ for (int f = 0; f < schema.getFieldCount(); f++) {
+ if (row.getFieldRepetitionCount(f) == 0) continue;
+ PrimitiveType.PrimitiveTypeName type =
+ schema.getType(f).asPrimitiveType().getPrimitiveTypeName();
+ if (type == PrimitiveType.PrimitiveTypeName.DOUBLE) {
+ row.getDouble(f, 0);
+ } else if (type == PrimitiveType.PrimitiveTypeName.FLOAT) {
+ row.getFloat(f, 0);
+ }
+ }
+ rowsDecoded++;
+ }
+ }
+ totalRows += rowsDecoded;
+ filesRead++;
+ }
+ }
+ assertEquals("Every column chunk should declare ALP", totalChunks, alpChunks);
+ LOG.info(
+ "readAllFixtureFilesIndependently: {} files, {} chunks ({} ALP), {} rows decoded",
+ filesRead, totalChunks, alpChunks, totalRows);
+ }
+
+ // ---------------------------------------------------------------------------
+ // Corner-case fixture per parquet-testing issue #105. Writes a single small
+ // file with one ALP-encoded column per corner case (no exceptions, 1 exception,
+ // all exceptions, NaN/Inf/-0.0, nulls, constant/bit_width=0, multi-vector with
+ // differing exponents) for both f32 and f64, then reads back and verifies
+ // every value bit-exactly against the expected pattern.
+ //
+ // The output file (alp_java_cornercases.parquet) is intended as a candidate
+ // fixture for parquet-testing PR #100 once Prateek confirms naming and design.
+ // ---------------------------------------------------------------------------
+
+ private static final int CORNER_ROWS_PER_VECTOR = 1024;
+ private static final int CORNER_VECTORS = 2;
+ private static final int CORNER_TOTAL_ROWS = CORNER_ROWS_PER_VECTOR * CORNER_VECTORS;
+
+ /** Expected values per corner-case column. Doubles use NaN to mark a null cell for sentinel use. */
+ private static class CornerCaseData {
+ double[] doubleVals;
+ float[] floatVals;
+ boolean[] isNull; // only meaningful for optional columns
+ boolean isFloat;
+ boolean isOptional;
+ String name;
+
+ static CornerCaseData doubles(String name, double[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.doubleVals = vals;
+ d.isFloat = false;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData floats(String name, float[] vals) {
+ CornerCaseData d = new CornerCaseData();
+ d.name = name;
+ d.floatVals = vals;
+ d.isFloat = true;
+ d.isOptional = false;
+ return d;
+ }
+
+ static CornerCaseData optionalDoubles(String name, double[] vals, boolean[] nulls) {
+ CornerCaseData d = doubles(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+
+ static CornerCaseData optionalFloats(String name, float[] vals, boolean[] nulls) {
+ CornerCaseData d = floats(name, vals);
+ d.isOptional = true;
+ d.isNull = nulls;
+ return d;
+ }
+ }
+
+ private List<CornerCaseData> buildCornerCaseColumns() {
+ List<CornerCaseData> cols = new ArrayList<>();
+ int N = CORNER_TOTAL_ROWS;
+
+ // 1. f64_no_exceptions: clean 2-decimal values, both vectors low bit-width
+ double[] noExc = new double[N];
+ for (int i = 0; i < N; i++) noExc[i] = (i % 1000) / 100.0;
+ cols.add(CornerCaseData.doubles("f64_no_exceptions", noExc));
+
+ // 2. f64_one_exception_per_vector: clean values with 1 NaN per 1024-row vector
+ double[] oneExc = new double[N];
+ for (int i = 0; i < N; i++) oneExc[i] = (i % 1000) / 100.0;
+ oneExc[100] = Double.NaN;
+ oneExc[1100] = Double.NaN;
+ cols.add(CornerCaseData.doubles("f64_one_exception_per_vector", oneExc));
+
+ // 3. f64_all_nan: every value NaN — all-exception vector
+ double[] allNan = new double[N];
+ java.util.Arrays.fill(allNan, Double.NaN);
+ cols.add(CornerCaseData.doubles("f64_all_nan", allNan));
+
+ // 4. f64_nan_inf_negzero: mix of special values across two vectors
+ double[] mixed = new double[N];
+ for (int i = 0; i < N; i++) mixed[i] = (i % 100) / 10.0;
+ mixed[0] = Double.NaN;
+ mixed[1] = Double.POSITIVE_INFINITY;
+ mixed[2] = Double.NEGATIVE_INFINITY;
+ mixed[3] = -0.0;
+ mixed[1024] = Double.NaN;
+ mixed[1025] = -0.0;
+ cols.add(CornerCaseData.doubles("f64_nan_inf_negzero", mixed));
+
+ // 5. f64_constant: every value = 42.0 → max_delta=0 → bit_width=0
+ double[] constant = new double[N];
+ java.util.Arrays.fill(constant, 42.0);
+ cols.add(CornerCaseData.doubles("f64_constant", constant));
+
+ // 6. f64_different_e_f_per_vector: vector 0 uses e=2,f=0 (cents); vector 1 uses e=3,f=0 (mils)
+ double[] diff = new double[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diff[i] = (i % 100) / 100.0;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diff[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0;
+ cols.add(CornerCaseData.doubles("f64_different_ef_per_vector", diff));
+
+ // 7. f64_with_nulls: optional column, ~30% nulls scattered
+ double[] withNulls = new double[N];
+ boolean[] nullMask = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMask[i] = true;
+ } else {
+ withNulls[i] = (i % 100) / 100.0;
+ }
+ }
+ cols.add(CornerCaseData.optionalDoubles("f64_with_nulls", withNulls, nullMask));
+
+ // 8–14: f32 analogues
+ float[] noExcF = new float[N];
+ for (int i = 0; i < N; i++) noExcF[i] = (i % 1000) / 100.0f;
+ cols.add(CornerCaseData.floats("f32_no_exceptions", noExcF));
+
+ float[] oneExcF = new float[N];
+ for (int i = 0; i < N; i++) oneExcF[i] = (i % 1000) / 100.0f;
+ oneExcF[100] = Float.NaN;
+ oneExcF[1100] = Float.NaN;
+ cols.add(CornerCaseData.floats("f32_one_exception_per_vector", oneExcF));
+
+ float[] allNanF = new float[N];
+ java.util.Arrays.fill(allNanF, Float.NaN);
+ cols.add(CornerCaseData.floats("f32_all_nan", allNanF));
+
+ float[] mixedF = new float[N];
+ for (int i = 0; i < N; i++) mixedF[i] = (i % 100) / 10.0f;
+ mixedF[0] = Float.NaN;
+ mixedF[1] = Float.POSITIVE_INFINITY;
+ mixedF[2] = Float.NEGATIVE_INFINITY;
+ mixedF[3] = -0.0f;
+ mixedF[1024] = Float.NaN;
+ mixedF[1025] = -0.0f;
+ cols.add(CornerCaseData.floats("f32_nan_inf_negzero", mixedF));
+
+ float[] constantF = new float[N];
+ java.util.Arrays.fill(constantF, 42.0f);
+ cols.add(CornerCaseData.floats("f32_constant", constantF));
+
+ float[] diffF = new float[N];
+ for (int i = 0; i < CORNER_ROWS_PER_VECTOR; i++) diffF[i] = (i % 100) / 100.0f;
+ for (int i = CORNER_ROWS_PER_VECTOR; i < N; i++) diffF[i] = ((i - CORNER_ROWS_PER_VECTOR) % 1000) / 1000.0f;
+ cols.add(CornerCaseData.floats("f32_different_ef_per_vector", diffF));
+
+ float[] withNullsF = new float[N];
+ boolean[] nullMaskF = new boolean[N];
+ for (int i = 0; i < N; i++) {
+ if (i % 3 == 0) {
+ nullMaskF[i] = true;
+ } else {
+ withNullsF[i] = (i % 100) / 100.0f;
+ }
+ }
+ cols.add(CornerCaseData.optionalFloats("f32_with_nulls", withNullsF, nullMaskF));
+
+ return cols;
+ }
+
+ /**
+ * Dumps the corner-case construction recipe to a CSV truth file alongside the parquet.
+ * The CSV is built directly from the expected values (not by reading the parquet back),
+ * which makes it suitable as independent ground truth for cross-language verifiers.
+ *
+ * <p>Conventions:
+ * <ul>
+ * <li>Header row matches column names.
+ * <li>Empty field marks a null cell in an optional column.
+ * <li>Special values use Java's standard toString output: {@code NaN}, {@code Infinity},
+ * {@code -Infinity}. These parse correctly via C++ {@code std::stod} / {@code std::stof}
+ * (per the standard, both are case-insensitive and accept the full word "Infinity").
+ * <li>{@code -0.0} round-trips through both Java and C++ string conversion bit-exact.
+ * </ul>
+ */
+ private void writeCornerCaseCsvTruth(java.nio.file.Path csvPath, List<CornerCaseData> cols) throws IOException {
+ try (java.io.BufferedWriter w = java.nio.file.Files.newBufferedWriter(csvPath)) {
+ // Header
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ w.write(cols.get(i).name);
+ }
+ w.write("\n");
+ // Data
+ for (int r = 0; r < CORNER_TOTAL_ROWS; r++) {
+ for (int i = 0; i < cols.size(); i++) {
+ if (i > 0) w.write(",");
+ CornerCaseData c = cols.get(i);
+ if (c.isOptional && c.isNull[r]) {
+ // empty field = null
+ } else if (c.isFloat) {
+ w.write(Float.toString(c.floatVals[r]));
+ } else {
+ w.write(Double.toString(c.doubleVals[r]));
+ }
+ }
+ w.write("\n");
+ }
+ }
+ }
+
+ @Test
+ public void generateAndVerifyCornerCaseFixture() throws IOException {
Review Comment:
Added coverage for this. There are wide_for_range columns in the corner-case fixture plus always-run testFloatWideForRange / testDoubleWideForRange unit tests that push the frame-of-reference minimum deeply negative and the FOR delta to a high bit width. One caveat: true INT/LONG-extreme encodings can't stay lossless for float (they'd turn into exceptions), so these cover the widest signed FOR range each type can hold without exceptions.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForFloat.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for FLOAT type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded float values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePacker-based unpacking.
+ */
+public class AlpValuesReaderForFloat extends AlpValuesReader {
+
+ private float[] decodedValues;
+ private int[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final int[] unpackPadBuf = new int[8];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForFloat() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new float[capacity];
+ this.deltasBuffer = new int[capacity];
+ this.excPositionsBuffer = new int[capacity];
+ this.unpackByteBuf = new byte[Integer.SIZE]; // max bit width for int = 32 bytes
+ }
+
+ @Override
+ public float readFloat() {
+ if (currentIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP float data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int indexInVector = currentIndex % vectorSize;
+ currentIndex++;
+ return decodedValues[indexInVector];
+ }
+
+ @Override
+ protected void decodeVector(int vectorIdx) {
+ int vectorLen = getVectorLength(vectorIdx);
+ int pos = getVectorDataPosition(vectorIdx);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = getShortLE(vectorsData, pos + 2) & 0xFFFF;
Review Comment:
Done. Lifted the shared vector-decode flow (ensureVectorDecoded/decodeVector, offset handling, exception patching) up into the base AlpValuesReader, so the float and double readers only implement decodeBody now.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
Review Comment:
Thanks!
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
+
+ // ---------------------------------------------------------------------------
+ // Helpers: build a known-good encoded page, then mutate copies of it
+ // ---------------------------------------------------------------------------
+
+ /** Build a valid ALP-encoded double page with N clean values. */
+ private static byte[] validDoublePage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(512, valueCount * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ // 2-decimal values — the ALP sweet spot, ensures no exceptions
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeDouble((i % 1000) / 100.0);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Build a valid ALP-encoded float page with N clean values. */
+ private static byte[] validFloatPage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(256, valueCount * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeFloat((i % 1000) / 100.0f);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Sanity baseline: the known-good page actually decodes cleanly. */
+ @Test
+ public void sanityBaselineDecodesClean() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Header-level validation (already-validated paths)
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsBadCompressionMode() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[0] = (byte) 0x99; // mode is at byte 0
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("compression"));
+ }
+
+ @Test
+ public void rejectsBadIntegerEncoding() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[1] = (byte) 0x99; // integer_encoding is at byte 1
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("integer encoding"));
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooLarge() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 99; // log_vector_size at byte 2
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("vector size"));
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooSmall() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 2; // below MIN_LOG_VECTOR_SIZE=3
+ assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ }
+
+ @Test
+ public void rejectsNegativeNumElements() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements is int32 LE at bytes 3..6 — write -1
+ page[3] = (byte) 0xFF;
+ page[4] = (byte) 0xFF;
+ page[5] = (byte) 0xFF;
+ page[6] = (byte) 0xFF;
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("element count"));
+ }
+
+ @Test
+ public void rejectsNumElementsGreaterThanValuesCount() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements stays 32; pass valuesCount=10 (smaller than encoded count)
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(10, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exceeds"));
+ }
+
+ // ---------------------------------------------------------------------------
+ // Vector-level validation (already-validated paths, surface lazily on decode)
+ // ---------------------------------------------------------------------------
+
+ /** Helper: find the byte position where the first vector's metadata starts. */
+ private static int firstVectorOffset(byte[] page) {
+ // header (7) + first 4 bytes of offset array = the offset value itself
+ int firstVectorOff = ByteBuffer.wrap(page, 7, 4).order(ByteOrder.LITTLE_ENDIAN).getInt();
+ // offsets are measured from the start of the compression body (after the 7B header)
+ return 7 + firstVectorOff;
+ }
+
+ @Test
+ public void rejectsExponentTooHighDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99; // exponent byte
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exponent"));
+ }
+
+ @Test
+ public void rejectsExponentTooHighFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("exponent"));
+ }
+
+ @Test
+ public void rejectsFactorGreaterThanExponent() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 2; // exponent
+ page[v0 + 1] = (byte) 5; // factor > exponent
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("factor"));
+ }
+
+ @Test
+ public void rejectsTooManyExceptions() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // num_exceptions at v0+2, uint16 LE — set to 9999, way more than vectorLen=16
+ page[v0 + 2] = (byte) (9999 & 0xFF);
+ page[v0 + 3] = (byte) ((9999 >>> 8) & 0xFF);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertTrue(ex.getMessage(), ex.getMessage().toLowerCase().contains("numexceptions"));
+ }
+
+ // ---------------------------------------------------------------------------
+ // Currently-unvalidated paths: truncation and corrupted offsets
+ // These currently fail with low-level Throwables (BufferUnderflowException,
+ // IndexOutOfBoundsException). The tests assert any Throwable is raised so we
+ // notice if a regression silently swallows the corruption.
+ // ---------------------------------------------------------------------------
+
+ /** Page with only the 7-byte header — nothing else. */
+ @Test
+ public void rejectsHeaderOnlyPage() {
+ byte[] tiny = new byte[]{0x00, 0x00, 0x0A, 0x20, 0x00, 0x00, 0x00}; // 32 elements, log_vec=10
Review Comment:
Yeah, it's valid. An all-null page has num_elements 0, so numVectors works out to 0 and decoding is a no-op. Added testFloatAllNullPage / testDoubleAllNullPage to cover it explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504140133
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,155 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private int[] excPositionsBuffer;
+ private final long[] unpackPadBuf = new long[8];
Review Comment:
can we make a constant for 8, and use it where appropriate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3566426181
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java:
##########
@@ -147,7 +148,13 @@ private ValuesWriter getInt96ValuesWriter(ColumnDescriptor path) {
private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) {
final ValuesWriter fallbackWriter;
- if (this.parquetProperties.isByteStreamSplitEnabled(path)) {
+ if (this.parquetProperties.isAlpEnabled(path)) {
Review Comment:
Good call on the tests. I added coverage in DefaultValuesWriterFactoryTest for both the float and double factories producing the ALP writers under V1 and V2, including the per-column case and ALP taking precedence over byte stream split.
On the placement question, I went back and forth but I'd like to keep ALP registered in both V1 and V2 for now. It's a fallback data-page encoding and doesn't rely on anything specific to the V2 page format, so there's no real reason a PARQUET_1_0 writer can't use it, and a couple of the interop tests do exercise ALP with V1 pages. It stays off by default either way, so nobody picks it up unless they explicitly call withAlpEncoding. If the new versioning work lands and we decide ALP should be pinned to a particular writer version, I'm happy to move it then.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
Review Comment:
Fair point, that wording is misleading. ALP isn't in the parquet-format spec yet (the linked PR is still the proposal), so there's no official byte layout to reference. In practice the Arrow C++ implementation is what everyone is matching against right now, which is why I called it out, but you're right that "C++ wire compatibility" makes it sound language-specific when it really isn't.
What we actually care about is that the encoded bytes match the reference layout, so a file written by one implementation reads back identically in another. I'll reword this to talk about matching the reference layout rather than singling out C++. Once the format PR merges we can point at the spec instead.
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
Yep, exactly. This skip is only here because ALP isn't in the parquet-format Encoding enum yet, so the round-trip check has nothing to compare against. Once the format change is released and we pick up the new enum value, the guard comes out and ALP goes through the same check as every other encoding. I left the note so we don't forget, and I'm tracking it against the format PR.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,178 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int currentIndex;
+ protected int currentVectorIndex;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ AlpValuesReader() {
+ this.currentIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorIndex = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.currentIndex = 0;
+ this.currentVectorIndex = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
Review Comment:
Right, that's the all-null case. When every value on the page is null nothing reaches the writer, so num_elements is 0, numVectors works out to 0, offsetArraySize is 0, and we end up calling stream.slice(0) with empty offset and vector buffers. Nothing gets decoded and the reader just reports 0 values, which is what we want, so it already handles it correctly.
I didn't have an explicit all-null test at the reader level, so I added testFloatAllNullPage and testDoubleAllNullPage to lock this in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504232644
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
this also needs to be removed once parquet-format is released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3508994249
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java:
##########
@@ -147,7 +148,13 @@ private ValuesWriter getInt96ValuesWriter(ColumnDescriptor path) {
private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) {
final ValuesWriter fallbackWriter;
- if (this.parquetProperties.isByteStreamSplitEnabled(path)) {
+ if (this.parquetProperties.isAlpEnabled(path)) {
Review Comment:
I don't see a test for this or the float factory producing the ALPValueWriter classes. Probably worth adding them just like the ByteStreamSplit tests in DefaultValuesWriterFactoryTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3959428444
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -51,6 +53,8 @@ public class ParquetProperties {
public static final int DEFAULT_DICTIONARY_PAGE_SIZE = DEFAULT_PAGE_SIZE;
public static final boolean DEFAULT_IS_DICTIONARY_ENABLED = true;
public static final boolean DEFAULT_IS_BYTE_STREAM_SPLIT_ENABLED = false;
+ public static final boolean DEFAULT_IS_ALP_ENABLED = false;
+ public static final int DEFAULT_ALP_VECTOR_SIZE = AlpConfig.DEFAULT_VECTOR_SIZE;
Review Comment:
Why rename this constant? Feels like we should just be calling AlpConfig.DEFAULT_VECTOR_SIZE at the call site
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3973397225
##########
parquet-hadoop/src/test/java/org/apache/parquet/format/converter/TestParquetMetadataConverter.java:
##########
@@ -471,6 +471,10 @@ public void testLogicalToConvertedTypeConversion() {
public void testEnumEquivalence() {
ParquetMetadataConverter parquetMetadataConverter = new ParquetMetadataConverter();
for (org.apache.parquet.column.Encoding encoding : org.apache.parquet.column.Encoding.values()) {
+ // Skip ALP encoding as it's not yet in the parquet-format specification
Review Comment:
I'd vote drop it and update our thrift using the update thrift script so we don't have the local addition. This is already merged upstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4068823424
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
Review Comment:
Please use AssertJ `assertThatThrownBy`. The test guide disallows JUnit assertion helpers. It seems that CI is complaining about the same thing:
```
[ERROR] Banned imports detected in TEST code:
[ERROR]
[ERROR] Reason: Use JUnit 5 (org.junit.jupiter.*) and AssertJ instead of JUnit 4
[ERROR] in file:///home/runner/work/parquet-java/parquet-java/parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInputFormatFileStatusReuse.java
[ERROR] static org.junit.Assert.assertEquals (Line: 20, Matched by: org.junit.**)
[ERROR] static org.junit.Assert.assertSame (Line: 21, Matched by: org.junit.**)
[ERROR] static org.junit.Assert.assertTrue (Line: 22, Matched by: org.junit.**)
[ERROR] org.junit.Rule (Line: 50, Matched by: org.junit.**)
[ERROR] org.junit.Test (Line: 51, Matched by: org.junit.**)
[ERROR] org.junit.rules.TemporaryFolder (Line: 52, Matched by: org.junit.**)
```
##########
parquet-column/src/test/java/org/apache/parquet/column/values/factory/DefaultValuesWriterFactoryTest.java:
##########
@@ -631,6 +668,50 @@ private void testFloatingPoint_WithByteStreamSplitAndDictionary(
PlainValuesWriter.class);
}
+ private void testFloatingPoint_WithAlp(
+ PrimitiveTypeName typeName, WriterVersion writerVersion, Class<? extends ValuesWriter> alpWriterClass) {
+ // Cross-column ALP without a dictionary: the ALP writer is produced directly.
+ doTestValueWriter(
+ createColumnDescriptor(typeName),
+ ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withDictionaryEncoding(false)
+ .withAlp()
+ .build(),
+ alpWriterClass);
+ // Cross-column ALP with a dictionary: the ALP writer is the dictionary's fallback.
+ doTestValueWriter(
+ createColumnDescriptor(typeName),
+ ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withAlp()
+ .build(),
+ DictionaryValuesWriter.class,
+ alpWriterClass);
+ // Per-column ALP: only colA opts in, colB keeps the plain fallback.
+ ParquetProperties properties = ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withDictionaryEncoding(false)
+ .withAlp("colA")
+ .build();
+ doTestValueWriter(createColumnDescriptor(typeName, "colA"), properties, alpWriterClass);
+ doTestValueWriter(createColumnDescriptor(typeName, "colB"), properties, PlainValuesWriter.class);
+ }
+
+ private void testAlpTakesPrecedenceOverByteStreamSplit(
+ PrimitiveTypeName typeName, WriterVersion writerVersion, Class<? extends ValuesWriter> alpWriterClass) {
+ // When both ALP and byte-stream-split are enabled the factory prefers ALP.
Review Comment:
This is obsolete. Production validation now rejects ALP and BYTE_STREAM_SPLIT on the same column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575347724
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
Review Comment:
Uses the signed range now: the estimator computes the FOR span as (long) maxEncoded - (long) minEncoded, so a mixed-sign vector no longer looks like a huge unsigned span.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
Review Comment:
Renamed to isIntrinsicFloatException (the blanket checks) and isFloatException (the full per-(e,f) check), so the subset relationship is clear from the names.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
Review Comment:
Dropped these one-liners since they only restated the code. The operation-order rationale that actually matters is in the class javadoc.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
Review Comment:
Same rename applied to the double side: isIntrinsicDoubleException / isDoubleException.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
Review Comment:
Removed the C++ cost-model note.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Integer.SIZE - Integer.numberOfLeadingZeros(maxDelta);
+ }
+
+ static int bitWidthForLong(long maxDelta) {
+ if (maxDelta == 0) {
+ return 0;
+ }
+ return Long.SIZE - Long.numberOfLeadingZeros(maxDelta);
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ /**
+ * Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the unsigned range of non-exception
+ * encoded values after frame-of-reference subtraction. This matches the C++ ALP cost model and
+ * produces better compression ratios than minimizing exception count alone.
+ */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ int bestExponent = 0;
+ int bestFactor = 0;
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int e = 0; e <= FLOAT_MAX_EXPONENT; e++) {
+ for (int f = 0; f <= e; f++) {
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
+ if (encoded < minEncoded) minEncoded = encoded;
+ if (encoded > maxEncoded) maxEncoded = encoded;
+ }
+ }
+ int nonExceptions = length - exceptions;
+ if (nonExceptions == 0) continue;
+ long delta = (nonExceptions < 2) ? 0 :
+ Integer.toUnsignedLong(maxEncoded) - Integer.toUnsignedLong(minEncoded);
+ int bitsPerValue = (delta == 0) ? 0 : (64 - Long.numberOfLeadingZeros(delta));
+ long estimatedSize = (long) length * bitsPerValue
+ + (long) exceptions * (Float.SIZE + Short.SIZE);
+ if (estimatedSize < bestEstimatedSize
Review Comment:
Documented it in the pickBestFloat javadoc now: smallest estimated size wins, ties break toward the higher exponent then the higher factor, and a zero-exception single-FOR pair wins outright.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-6485786354
@vinooganesh Could you please rebase again to make ci happy?
@rdblue Do you want to take a look again? It seems that all your feedbacks have been addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub) via issues" <is...@parquet.apache.org>.
wgtmac merged PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4098979310
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
Review Comment:
Instead of a no-arg constructor that initializes mutable state, I think we generally prefer initializing where the fields are declared:
```java
protected int pageValueIndex = 0;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099384678
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
Review Comment:
I think this method should be static and should pass in the offset array and data size. As it is written now, there isn't much of a distinction between what is passed as an argument vs accessed as a field. Looks like it uses fields, but passes local variables (even when they can be derived).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099674094
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
Review Comment:
I expected the "body" to mean the vector's "Data Section" (using the term from the spec) but instead it looks like this decodes the packed integer values, since the exception positions and exception values are decoded below in this method. A better name would help here, but I would also like to see the logic of decoding broken down along the same lines as the spec, and for there to be more use of helper methods to keep function size and complexity small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3862: Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4305080281
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
Review Comment:
Renamed it to `totalValueCount`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
Review Comment:
Yep, renamed. I also went through the rest of the reader so the indexes are `vectorIndex`, `pageValueIndex`, and `vectorValueIndex` throughout.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
Review Comment:
Split it up into `readPageHeader`, `readOffsetArray`, `sliceVectorData`, and `validateVectorOffsets`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
Review Comment:
Changed to index, here and everywhere else I found it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
Review Comment:
Moved it below the two position methods, made it static, and added javadoc. I went with `getVectorSize` rather than `vectorSize` so it matches `getVectorDataPosition` and `getVectorEndPosition`, but I can drop the `get` if you prefer.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
Review Comment:
Added a static `isLast(vectorIndex, numVectors)` and used it in `getVectorSize`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+
+ // Bound the vector count against the bytes actually present before allocating anything.
+ // numElements is only capped by valuesCount, which a forged page can set as high as
+ // Integer.MAX_VALUE, so computing this in int would overflow to a negative count, and even
+ // without overflow a large count would drive a huge int[] and a huge stream.slice allocation.
+ // Every vector costs a 4-byte offset entry plus at least its ALP and FOR headers, which gives
+ // a hard ceiling on how many vectors the remaining bytes can possibly describe.
+ int minVectorBytes = ALP_INFO_SIZE + forInfoSize();
+ long bodyBytes = stream.available();
+ long numVectorsLong = ((long) numElements + vectorSize - 1) / vectorSize;
+ long minBodyBytes = numVectorsLong * (Integer.BYTES + minVectorBytes);
+ if (minBodyBytes > bodyBytes) {
+ throw new ParquetDecodingException("ALP header element count " + numElements + " requires at least "
+ + minBodyBytes + " bytes but only " + bodyBytes + " remain in the page");
+ }
+ this.numVectors = (int) numVectorsLong;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ validateVectorOffsets(minVectorBytes);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ /**
+ * Validates the offset array against the page body before any vector is decoded. Offsets are
+ * relative to the start of the compression body (the offset array itself), so the first vector
+ * must begin exactly where the offset array ends, offsets must increase by at least one vector's
+ * fixed headers, and every vector must start early enough to fit those headers. Without this a
+ * forged offset can point into another vector and silently decode the wrong values.
+ */
+ private void validateVectorOffsets(int minVectorBytes) {
+ int bodyLimit = offsetArraySize + vectorsData.limit();
+ for (int v = 0; v < numVectors; v++) {
+ int offset = vectorOffsets[v];
+ if (v == 0) {
+ if (offset != offsetArraySize) {
+ throw new ParquetDecodingException("ALP first vector offset " + offset
+ + " must equal the offset array size " + offsetArraySize);
+ }
+ } else if ((long) offset - vectorOffsets[v - 1] < minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector offsets must increase by at least "
+ + minVectorBytes + " bytes, but vector " + v + " starts at " + offset + " after vector "
+ + (v - 1) + " at " + vectorOffsets[v - 1]);
+ }
+ if (offset > bodyLimit - minVectorBytes) {
+ throw new ParquetDecodingException("ALP vector " + v + " offset " + offset
+ + " leaves no room for its headers before the end of the page body at " + bodyLimit);
+ }
+ }
+ }
+
+ /**
+ * The position just past the last byte vector {@code vectorNumber} may read, derived from the
+ * next vector's offset (or the end of the body for the final vector).
+ */
+ private int getVectorEndPosition(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorOffsets[vectorNumber + 1] - offsetArraySize;
+ }
+ return vectorsData.limit();
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ // Compare against the remaining count rather than adding to pageValueIndex: a large n would
+ // overflow the sum, pass the check, and leave pageValueIndex negative.
+ if (n < 0 || n > totalCount - pageValueIndex) {
+ throw new ParquetDecodingException(String.format(
+ "Cannot skip this many elements. Current index: %d. Skip %d. Total count: %d",
+ pageValueIndex, n, totalCount));
+ }
+ pageValueIndex += n;
+ }
+
+ protected void ensureVectorDecoded() {
+ int vectorNumber = pageValueIndex / vectorSize;
+ if (vectorNumber != currentVectorNumber) {
+ decodeVector(vectorNumber);
+ currentVectorNumber = vectorNumber;
+ }
+ }
+
+ /**
+ * Decodes one vector: parses and validates the shared ALP/FOR header fields, delegates the
+ * type-specific numeric work (FOR read, bit-unpacking, decode loop, exception values) to hooks,
+ * and reads the shared exception-position list in between. Runs once per vector, so the virtual
+ * hook calls are off the per-value hot path.
+ */
+ protected final void decodeVector(int vectorNumber) {
+ int vectorLen = getVectorLength(vectorNumber);
+ int pos = getVectorDataPosition(vectorNumber);
+
+ int exponent = vectorsData.get(pos) & 0xFF;
+ int factor = vectorsData.get(pos + 1) & 0xFF;
+ int numExceptions = vectorsData.getShort(pos + 2) & 0xFFFF;
+ pos += ALP_INFO_SIZE;
+
+ if (exponent > maxExponent()) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " exponent " + exponent + " in vector "
+ + vectorNumber + ", max is " + maxExponent());
+ }
+ if (factor > exponent) {
+ throw new ParquetDecodingException("Invalid ALP " + typeName() + " factor " + factor + " > exponent "
+ + exponent + " in vector " + vectorNumber);
+ }
+ if (numExceptions > vectorLen) {
+ throw new ParquetDecodingException("Invalid ALP numExceptions " + numExceptions + " > vectorLen "
+ + vectorLen + " in vector " + vectorNumber);
+ }
+
+ pos = decodeBody(pos, vectorNumber, vectorLen, exponent, factor);
Review Comment:
Renamed and restructured. `decodeVector` now reads the vector header (`AlpInfo`, then `ForInfo`) and then the data section in the spec's order: packed values, exception positions, exception values. There's no more "body".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3611811910
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
+ if (maxDelta == 0) {
Review Comment:
That was bitWidthForInt, the local int bit-width helper. It isn't there to restate the spec: I removed it in favor of BytesUtils.getWidthFromMaxInt (used in the float writer). Only bitWidthForLong stays, because BytesUtils has no long version and the double writer needs one; its javadoc now notes it's the long counterpart to getWidthFromMaxInt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3575377720
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Check round-trip: encode then decode, and see if we get the same bits back. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ static boolean isDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ /** Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) — single expression. */
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ /** Decode: encoded * POW10[f] * POW10_NEGATIVE[e] — single expression. */
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ /** Number of bits needed to represent maxDelta as an unsigned value. */
+ static int bitWidthForInt(int maxDelta) {
Review Comment:
Correction to my earlier reply, to be precise about where each is used: BytesUtils.getWidthFromMaxInt is int-only, so the float writer's FOR bit width uses it directly. The double writer uses AlpEncoderDecoder.bitWidthForLong (the long counterpart, since BytesUtils has no long version). The size estimator inlines the same long bit-width math (64 - Long.numberOfLeadingZeros(delta)) rather than calling either helper. My earlier reply wrongly attributed bitWidthForLong to the estimator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3504151483
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1066 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assume.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
Review Comment:
we've also tested the other way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3503976300
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Uses multiply-by-reciprocal (via POW10_NEGATIVE) for C++ wire compatibility.
Review Comment:
C++ wire compatibility doesn't make sense to me here, why is it tied to a particular language. Is this something that is missing from the spec for numeric stability, what would the alternative be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4089197573
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,398 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
Review Comment:
Done in de1e6764e. Every JUnit assertion and assumption helper is gone from the ALP tests: `assertThrows` became `assertThatThrownBy` in `AlpAdversarialTest` and `AlpValuesEndToEndTest`, and `assumeTrue` became AssertJ's `assumeThat` in `TestInterOpReadAlp`.
While converting I also pinned the real exception messages rather than the lowercased substring the old assertions used. That turned out to matter: the old `contains("exponent")` check would have passed on a *factor* error too, since that message mentions the exponent as well.
`catchClean` became `assertFailsCleanly`, built on `assertThatThrownBy(...).isInstanceOfAny(...)`. I checked it still holds the property you were after in your earlier comment: it fails when nothing is thrown, when the wrong type is thrown, and when an `OutOfMemoryError` escapes, so an allocation bomb still cannot pass as a clean rejection.
To convince myself the new message assertions were load-bearing rather than decorative, I broke four validations in `AlpValuesReader` in turn — the factor check, the skip bounds, the non-increasing offset check, and the wording of the allocation-bound message — and confirmed each one is caught by exactly the converted test.
On the CI output you pasted: that one is not from this PR. It is `TestInputFormatFileStatusReuse`, which arrived with #3674 and was migrated to JUnit 5 upstream in #3813. Merging master picks up the fix, and parquet-hadoop's banned-imports check passes on the branch now.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/factory/DefaultValuesWriterFactoryTest.java:
##########
@@ -631,6 +668,50 @@ private void testFloatingPoint_WithByteStreamSplitAndDictionary(
PlainValuesWriter.class);
}
+ private void testFloatingPoint_WithAlp(
+ PrimitiveTypeName typeName, WriterVersion writerVersion, Class<? extends ValuesWriter> alpWriterClass) {
+ // Cross-column ALP without a dictionary: the ALP writer is produced directly.
+ doTestValueWriter(
+ createColumnDescriptor(typeName),
+ ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withDictionaryEncoding(false)
+ .withAlp()
+ .build(),
+ alpWriterClass);
+ // Cross-column ALP with a dictionary: the ALP writer is the dictionary's fallback.
+ doTestValueWriter(
+ createColumnDescriptor(typeName),
+ ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withAlp()
+ .build(),
+ DictionaryValuesWriter.class,
+ alpWriterClass);
+ // Per-column ALP: only colA opts in, colB keeps the plain fallback.
+ ParquetProperties properties = ParquetProperties.builder()
+ .withWriterVersion(writerVersion)
+ .withDictionaryEncoding(false)
+ .withAlp("colA")
+ .build();
+ doTestValueWriter(createColumnDescriptor(typeName, "colA"), properties, alpWriterClass);
+ doTestValueWriter(createColumnDescriptor(typeName, "colB"), properties, PlainValuesWriter.class);
+ }
+
+ private void testAlpTakesPrecedenceOverByteStreamSplit(
+ PrimitiveTypeName typeName, WriterVersion writerVersion, Class<? extends ValuesWriter> alpWriterClass) {
+ // When both ALP and byte-stream-split are enabled the factory prefers ALP.
Review Comment:
You're right, thank you. Removed in de1e6764e, along with both `Alp_TakesPrecedenceOverByteStreamSplit` tests and the shared helper. Since `validateAlp` rejects ALP and BYTE_STREAM_SPLIT on the same column in `newColumnWriteStore`, the factory's ordering can never be reached in production, so the test was asserting behaviour that cannot happen.
One thing worth noting: there was no dedicated precedence *code* to remove alongside it. `DefaultV1ValuesWriterFactory` just has a plain `if (isAlpEnabled) / else if (isByteStreamSplitEnabled)` chain, so ALP-first is an artifact of the ordering rather than a feature, and nothing is left dead by the deletion.
The rejection itself is covered by `TestParquetProperties.alp_withByteStreamSplitOnTheSameColumn_fails`, which builds the identical configuration the deleted test used.
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,775 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.column.values.alp.AlpConfig;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility tests for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads the ALP conformance file from parquet-testing and checks that every ALP column decodes
+ * bit-identically to its PLAIN counterpart, and round-trips Java-written ALP files.
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String ALP_EXTENDED_FILE = "alp_extended.zstd.parquet";
+ private static final String CHANGESET = "09f3cdb";
+ private final InterOpTester interop = new InterOpTester();
+
+ private static final int EXTENDED_ROWS = 9032;
+ private static final int EXTENDED_ROW_GROUPS = 5;
+ private static final int EXTENDED_NULLS = 8;
+
+ /** ALP column -> PLAIN column holding the same values. */
+ private static final Map<String, String> ALP_TO_PLAIN = new LinkedHashMap<>();
+
+ static {
+ ALP_TO_PLAIN.put("float_alp_32", "float_plain");
+ ALP_TO_PLAIN.put("float_alp_1024", "float_plain");
+ ALP_TO_PLAIN.put("float_alp_4096", "float_plain");
+ ALP_TO_PLAIN.put("double_alp_32", "double_plain");
+ ALP_TO_PLAIN.put("double_alp_1024", "double_plain");
+ ALP_TO_PLAIN.put("double_alp_4096", "double_plain");
+ }
+
+ private java.nio.file.Path getExtendedFile() throws IOException {
+ return Paths.get(interop.GetInterOpFile(ALP_EXTENDED_FILE, CHANGESET).toString());
+ }
+
+ /** Read all rows from a parquet file using LocalInputFile (no Hadoop FileSystem). */
+ private List<Group> readAllRows(java.nio.file.Path filePath) throws IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(filePath))) {
+ ParquetMetadata footer = reader.getFooter();
+ MessageType schema = footer.getFileMetaData().getSchema();
+ MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
+ PageReadStore pages;
+ while ((pages = reader.readNextRowGroup()) != null) {
+ long rowCount = pages.getRowCount();
+ RecordReader<Group> recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
+ for (long i = 0; i < rowCount; i++) {
+ rows.add(recordReader.read());
+ }
+ }
+ }
+ return rows;
+ }
+
+ private static boolean isNull(Group row, String field) {
+ return row.getFieldRepetitionCount(field) == 0;
+ }
+
+ /** Raw IEEE 754 bits of a float or double field, widened to a long for exact comparison. */
+ private static long rawBits(Group row, String field, boolean isFloat) {
+ return isFloat
+ ? Float.floatToRawIntBits(row.getFloat(field, 0)) & 0xFFFFFFFFL
+ : Double.doubleToRawLongBits(row.getDouble(field, 0));
+ }
+
+ @Test
+ public void testAlpColumnsMatchPlainColumns() throws IOException {
+ List<Group> rows = readAllRows(getExtendedFile());
+ assertThat(rows.size()).as("row count").isEqualTo(EXTENDED_ROWS);
+
+ for (Map.Entry<String, String> entry : ALP_TO_PLAIN.entrySet()) {
+ String alpCol = entry.getKey();
+ String plainCol = entry.getValue();
+ boolean isFloat = alpCol.startsWith("float");
+ int nulls = 0;
+ for (int i = 0; i < rows.size(); i++) {
+ Group row = rows.get(i);
+ boolean alpNull = isNull(row, alpCol);
+ assertThat(alpNull).as("nullness of " + alpCol + " at row " + i).isEqualTo(isNull(row, plainCol));
+ if (alpNull) {
+ nulls++;
+ continue;
+ }
+ long alpBits = rawBits(row, alpCol, isFloat);
+ long plainBits = rawBits(row, plainCol, isFloat);
+ assertThat(alpBits)
+ .as(String.format(
+ "%s row %d: bits 0x%x, %s bits 0x%x", alpCol, i, alpBits, plainCol, plainBits))
+ .isEqualTo(plainBits);
+ }
+ assertThat(nulls).as(alpCol + " nulls").isEqualTo(EXTENDED_NULLS);
+ }
+ }
+
+ @Test
+ public void testAlpColumnsAreAlpEncoded() throws IOException {
+ try (ParquetFileReader reader = ParquetFileReader.open(new LocalInputFile(getExtendedFile()))) {
+ ParquetMetadata footer = reader.getFooter();
+ assertThat(footer.getBlocks().size()).as("row groups").isEqualTo(EXTENDED_ROW_GROUPS);
+ int alpChunks = 0;
+ for (org.apache.parquet.hadoop.metadata.BlockMetaData block : footer.getBlocks()) {
+ for (org.apache.parquet.hadoop.metadata.ColumnChunkMetaData chunk : block.getColumns()) {
+ String path = chunk.getPath().toDotString();
+ boolean hasAlp = chunk.getEncodings().contains(Encoding.ALP);
+ assertThat(hasAlp)
+ .as("ALP on " + path + ": " + chunk.getEncodings())
+ .isEqualTo(ALP_TO_PLAIN.containsKey(path));
+ if (hasAlp) {
+ alpChunks++;
+ }
+ }
+ }
+ assertThat(alpChunks).as("ALP column chunks").isEqualTo(ALP_TO_PLAIN.size() * EXTENDED_ROW_GROUPS);
+ }
+ }
+
+ /**
+ * Checks exact bit patterns at known rows, so a fault that corrupted the ALP and PLAIN paths the
+ * same way would still be caught. NaN payloads must survive, since exceptions are stored verbatim.
+ */
+ @Test
+ public void testSpecialValueBitPatterns() throws IOException {
+ List<Group> rows = readAllRows(getExtendedFile());
+ assertBits(rows, 1024, 0x7FF8000000000000L, 0x7FC00000L, "canonical NaN");
+ assertBits(rows, 1500, 0x7FF800DEADBEEF00L, 0x7FC0DEADL, "NaN with payload");
+ assertBits(rows, 2047, 0xFFF8000000000001L, 0xFFC00001L, "negative NaN with payload");
+ assertBits(rows, 2000, 0x7FF0000000000000L, 0x7F800000L, "+Inf");
+ assertBits(rows, 2001, 0xFFF0000000000000L, 0xFF800000L, "-Inf");
+ assertBits(rows, 2002, 0x8000000000000000L, 0x80000000L, "-0.0");
+ assertBits(rows, 2003, 0x1L, 0x1L, "smallest subnormal");
+ assertBits(rows, 2500, 0x400921FB54442D18L, 0x40490FDBL, "pi");
+ assertBits(rows, 7168, 0x401F147AE147AE14L, 0x40F8A3D7L, "constant 7.77");
+ assertBits(rows, 9000, 0xC3DBC16D674EC800L, 0xDEDE0B6BL, "-8e18");
+ assertBits(rows, 9001, 0x43DBC16D674EC800L, 0x5EDE0B6BL, "8e18");
+ for (int row : new int[] {8200, 8900}) {
+ for (String col : ALP_TO_PLAIN.keySet()) {
+ assertThat(isNull(rows.get(row), col))
+ .as(col + " null at row " + row)
+ .isTrue();
+ }
+ }
+ }
+
+ private void assertBits(List<Group> rows, int row, long doubleBits, long floatBits, String what) {
+ for (String col : ALP_TO_PLAIN.keySet()) {
+ boolean isFloat = col.startsWith("float");
+ assertThat(rawBits(rows.get(row), col, isFloat))
+ .as(what + " at row " + row + " in " + col)
+ .isEqualTo(isFloat ? floatBits : doubleBits);
+ }
+ }
+
+ private static final String ALP_SCHEMA =
+ "message alp_interop { " + "required double double_col; " + "required float float_col; " + "}";
+
+ private static final double[] DOUBLE_VALUES = {
+ 1.23, 4.56, 7.89, 0.001, 1000.0, -3.14, 2.718281828, 9.99999, 0.123456789, 100.5
+ };
+ private static final float[] FLOAT_VALUES = {
+ 1.23f, 4.56f, 7.89f, 0.001f, 1000.0f, -3.14f, 2.718f, 9.999f, 0.1234f, 100.5f
+ };
+
+ /**
+ * Write an ALP-encoded file from Java using the given page version, then read it back and verify
+ * all double and float values round-trip exactly.
+ */
+ private void writeAndVerifyAlpFile(WriterVersion version) throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ java.nio.file.Path outPath =
+ newFolder().toPath().resolve("alp_java_" + version.name().toLowerCase() + ".parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(version)
+ .withAlp()
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", DOUBLE_VALUES[i]);
+ row.add("float_col", FLOAT_VALUES[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch for " + version).isEqualTo(DOUBLE_VALUES.length);
+ for (int i = 0; i < DOUBLE_VALUES.length; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i + " for " + version)
+ .isEqualTo(DOUBLE_VALUES[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i + " for " + version)
+ .isEqualTo(FLOAT_VALUES[i]);
+ }
+ LOG.info(
+ "writeAndVerifyAlpFile [{}]: wrote and read back {} rows from {}",
+ version,
+ rows.size(),
+ outPath.getFileName());
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V1 (PARQUET_1_0) data pages and reads them back.
+ * Verifies the Java write path produces a valid file readable by this implementation.
+ */
+ @Test
+ public void testJavaWriteAlpV1Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_1_0);
+ }
+
+ /**
+ * Writes >4096 rows with vectorSize=4096 so multiple full vectors are flushed, then verifies
+ * the file round-trips exactly. The reader pulls log_vector_size from the on-disk header to
+ * size its unpacking window, so a wrong header byte would surface as decode garbage —
+ * round-trip equality is sufficient proof that the configured vector size took effect.
+ */
+ @Test
+ public void testJavaWriteAlpCustomVectorSize() throws IOException {
+ MessageType schema = MessageTypeParser.parseMessageType(ALP_SCHEMA);
+ int rowCount = 4500; // crosses one full vector + partial tail at vectorSize=4096
+ double[] doubles = new double[rowCount];
+ float[] floats = new float[rowCount];
+ // 2-decimal sensor-like data — the ALP sweet spot, so few/no exceptions
+ for (int i = 0; i < rowCount; i++) {
+ doubles[i] = (i * 13L % 100000) / 100.0;
+ floats[i] = (float) ((i * 7L % 10000) / 100.0);
+ }
+
+ java.nio.file.Path outPath = newFolder().toPath().resolve("alp_java_vs4096.parquet");
+
+ try (ParquetWriter<Group> writer = ExampleParquetWriter.builder(new LocalOutputFile(outPath))
+ .withType(schema)
+ .withCompressionCodec(CompressionCodecName.UNCOMPRESSED)
+ .withWriterVersion(WriterVersion.PARQUET_2_0)
+ .withAlp(new AlpConfig(4096))
+ .withDictionaryEncoding(false)
+ .withConf(new Configuration())
+ .build()) {
+ for (int i = 0; i < rowCount; i++) {
+ SimpleGroup row = new SimpleGroup(schema);
+ row.add("double_col", doubles[i]);
+ row.add("float_col", floats[i]);
+ writer.write(row);
+ }
+ }
+
+ List<Group> rows = readAllRows(outPath);
+ assertThat(rows.size()).as("Row count mismatch at vectorSize=4096").isEqualTo(rowCount);
+ for (int i = 0; i < rowCount; i++) {
+ assertThat(rows.get(i).getDouble("double_col", 0))
+ .as("double_col mismatch at row " + i)
+ .isEqualTo(doubles[i]);
+ assertThat(rows.get(i).getFloat("float_col", 0))
+ .as("float_col mismatch at row " + i)
+ .isEqualTo(floats[i]);
+ }
+ LOG.info("testJavaWriteAlpCustomVectorSize: {} rows round-tripped at vectorSize=4096", rowCount);
+ }
+
+ /**
+ * Java writes ALP-encoded floats/doubles using V2 (PARQUET_2_0) data pages and reads them back.
+ */
+ @Test
+ public void testJavaWriteAlpV2Pages() throws IOException {
+ writeAndVerifyAlpFile(WriterVersion.PARQUET_2_0);
+ }
+
+ /**
+ * Writes ALP-encoded V1 and V2 parquet files from Java and verifies that pyarrow (Python Arrow)
+ * can read them back correctly. Skipped if python3/pyarrow is not available in the environment.
+ *
+ * <p>This test addresses cross-language compatibility: Java writes, another implementation reads.
+ */
+ @Test
+ public void testJavaWrittenAlpFilesReadableByPyarrow() throws IOException, InterruptedException {
Review Comment:
Agreed, removed in b40a5d761. It was hacky: it shelled out to `python3`, built its assertions by string-concatenating a Python script, and then skipped itself if pyarrow could not read ALP, so in practice it never ran anywhere.
One thing I would rather say out loud than let pass quietly. The C++ interop test and `testAlpColumnsMatchPlainColumns` both cover the inbound direction, C++ writes and Java reads. The pyarrow test was nominally the outbound check, Java writes and another implementation reads. Deleting it costs no CI coverage, since it never ran, but it does mean there is no automated Java-to-C++ check in the repo now.
I verified that direction by hand in July against a local Arrow build with ALP support (apache/arrow#48345): it recognised `Encodings: ALP`, decoded cleanly, and all 2048 rows of the extreme-value columns matched the Java output bit for bit. Happy to wire it up properly once the Arrow side is released, if you think it is worth having.
`TestInterOpReadAlp` now runs all 12 of its tests with nothing skipped.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpExceptionCountTest.java:
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Measures exception counts and compression ratios using the real Spotify and Arade datasets from
+ * the parquet-testing repository (Pratik's alpFloatingPointDataset branch).
+ *
+ * <p>These are the same datasets used in the meeting discussion to compare Java vs C++ ALP
+ * exception counts. Run from the project root with the alp-test-data/ directory present:
+ *
+ * <pre>
+ * ./mvnw test -pl parquet-column -Dtest=AlpExceptionCountTest
+ * </pre>
+ *
+ * <p>To use a different data directory:
+ *
+ * <pre>
+ * ./mvnw test -pl parquet-column -Dtest=AlpExceptionCountTest \
+ * -DALP_TEST_DATA_DIR=/path/to/alp-test-data
+ * </pre>
+ */
+public class AlpExceptionCountTest {
Review Comment:
Agreed, and removed in b40a5d761. You are right that it stopped earning its place: it needed an `alp-test-data` directory that is not in the repo, so all three tests skipped everywhere, including locally for anyone who had not fetched the datasets by hand.
It was genuinely useful while I was checking exception rates against the numbers in the ALP paper, but that is a one-off exercise rather than something worth keeping in the suite.
With this and the pyarrow test gone, parquet-column runs 882 tests with nothing skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-5806796789
Could you please create a GH issue and link it to the PR title to follow the convention of this repo? I don't have other comment before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099751081
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReaderForDouble.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_FOR_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.DOUBLE_MAX_EXPONENT;
+import static org.apache.parquet.column.values.alp.AlpConstants.PACK_GROUP_SIZE;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * ALP values reader for DOUBLE type with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded double values from the interleaved page layout.
+ * Each vector is decoded on first access using BytePackerForLong-based unpacking.
+ */
+public class AlpValuesReaderForDouble extends AlpValuesReader {
+
+ private double[] decodedValues;
+ private long[] deltasBuffer;
+ private final long[] unpackPadBuf = new long[PACK_GROUP_SIZE];
+ private byte[] unpackByteBuf;
+
+ public AlpValuesReaderForDouble() {
+ super();
+ }
+
+ @Override
+ protected void allocateDecodedBuffer(int capacity) {
+ this.decodedValues = new double[capacity];
+ this.deltasBuffer = new long[capacity];
+ this.unpackByteBuf = new byte[Long.SIZE]; // max bit width for long = 64 bytes
+ }
+
+ @Override
+ protected int maxExponent() {
+ return DOUBLE_MAX_EXPONENT;
+ }
+
+ @Override
+ protected String typeName() {
+ return "double";
+ }
+
+ @Override
+ protected int forInfoSize() {
+ return DOUBLE_FOR_INFO_SIZE;
+ }
+
+ @Override
+ protected int exceptionValueSize() {
+ return Double.BYTES;
+ }
+
+ @Override
+ public double readDouble() {
+ if (pageValueIndex >= totalCount) {
+ throw new ParquetDecodingException("ALP double data was already exhausted.");
+ }
+ ensureVectorDecoded();
+ int vectorSlot = pageValueIndex % vectorSize;
+ pageValueIndex++;
+ return decodedValues[vectorSlot];
+ }
+
+ @Override
+ protected int decodeBody(int pos, int vectorNumber, int vectorLen, int exponent, int factor) {
+ long frameOfReference = vectorsData.getLong(pos);
+ int bitWidth = vectorsData.get(pos + Long.BYTES) & 0xFF;
+ if (bitWidth > Long.SIZE) {
+ throw new ParquetDecodingException(
+ "Invalid ALP double bitWidth " + bitWidth + " > 64 in vector " + vectorNumber);
+ }
+ pos += DOUBLE_FOR_INFO_SIZE;
+
+ if (bitWidth > 0) {
+ pos = unpackLongsWithBytePacker(vectorsData, pos, deltasBuffer, vectorLen, bitWidth);
+ } else {
+ Arrays.fill(deltasBuffer, 0, vectorLen, 0L);
+ }
+
+ for (int i = 0; i < vectorLen; i++) {
+ long encoded = deltasBuffer[i] + frameOfReference;
+ decodedValues[i] = AlpCodec.decodeDouble(encoded, exponent, factor);
+ }
+ return pos;
+ }
+
+ @Override
+ protected void applyExceptionValues(int pos, int numExceptions) {
Review Comment:
This is another case where I think this would be better if it were static. It's strange to have the number of exceptions passed in as an argument, but the exception positions read directly from the parent's field. I think I would also move reading exception positions into this method (by calling another static helper) so that the dependency is clearer and the logic here is more self-contained.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3900437837
##########
parquet-format-structures/pom.xml:
##########
@@ -66,6 +66,35 @@
</execution>
</executions>
</plugin>
+ <!--
Review Comment:
We should probably remove these lines by either releasing a new version of parquet-format, or after merging https://github.com/apache/parquet-java/pull/3709
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
Review Comment:
This int arithmetic can overflow for a forged numElements. The result can be negative or trigger a huge allocation.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
Review Comment:
With the default 20K-row page limit, this cadence yields only about two samples per page, while eight are required. The preset cache never builds on normal pages.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
Review Comment:
These offsets are trusted without validating the first entry, ordering, or bounds. Decode also ignores the next offset, so malformed bytes can silently produce the wrong vector.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
+
+ // ---------------------------------------------------------------------------
+ // Helpers: build a known-good encoded page, then mutate copies of it
+ // ---------------------------------------------------------------------------
+
+ /** Build a valid ALP-encoded double page with N clean values. */
+ private static byte[] validDoublePage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(512, valueCount * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ // 2-decimal values — the ALP sweet spot, ensures no exceptions
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeDouble((i % 1000) / 100.0);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Build a valid ALP-encoded float page with N clean values. */
+ private static byte[] validFloatPage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(256, valueCount * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeFloat((i % 1000) / 100.0f);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Sanity baseline: the known-good page actually decodes cleanly. */
+ @Test
+ public void sanityBaselineDecodesClean() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Header-level validation (already-validated paths)
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsBadCompressionMode() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[0] = (byte) 0x99; // mode is at byte 0
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("compression"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBadIntegerEncoding() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[1] = (byte) 0x99; // integer_encoding is at byte 1
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("integer encoding"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooLarge() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 99; // log_vector_size at byte 2
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("vector size"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooSmall() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 2; // below MIN_LOG_VECTOR_SIZE=3
+ assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ }
+
+ @Test
+ public void rejectsNegativeNumElements() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements is int32 LE at bytes 3..6 — write -1
+ page[3] = (byte) 0xFF;
+ page[4] = (byte) 0xFF;
+ page[5] = (byte) 0xFF;
+ page[6] = (byte) 0xFF;
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("element count"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsNumElementsGreaterThanValuesCount() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements stays 32; pass valuesCount=10 (smaller than encoded count)
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(10, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("exceeds"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Vector-level validation (already-validated paths, surface lazily on decode)
+ // ---------------------------------------------------------------------------
+
+ /** Helper: find the byte position where the first vector's metadata starts. */
+ private static int firstVectorOffset(byte[] page) {
+ // header (7) + first 4 bytes of offset array = the offset value itself
+ int firstVectorOff =
+ ByteBuffer.wrap(page, 7, 4).order(ByteOrder.LITTLE_ENDIAN).getInt();
+ // offsets are measured from the start of the compression body (after the 7B header)
+ return 7 + firstVectorOff;
+ }
+
+ @Test
+ public void rejectsExponentTooHighDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99; // exponent byte
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsExponentTooHighFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsFactorGreaterThanExponent() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 2; // exponent
+ page[v0 + 1] = (byte) 5; // factor > exponent
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("factor"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsTooManyExceptions() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // num_exceptions at v0+2, uint16 LE — set to 9999, way more than vectorLen=16
+ page[v0 + 2] = (byte) (9999 & 0xFF);
+ page[v0 + 3] = (byte) ((9999 >>> 8) & 0xFF);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("numexceptions"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(8) then bitWidth byte at v0+12. 99 > 64.
+ page[v0 + 12] = (byte) 99;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(4) then bitWidth byte at v0+8. 99 > 32.
+ page[v0 + 8] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Currently-unvalidated paths: truncation and corrupted offsets
+ // These currently fail with low-level Throwables (BufferUnderflowException,
+ // IndexOutOfBoundsException). The tests assert any Throwable is raised so we
+ // notice if a regression silently swallows the corruption.
+ // ---------------------------------------------------------------------------
+
+ /** Page with only the 7-byte header — nothing else. */
+ @Test
+ public void rejectsHeaderOnlyPage() {
+ byte[] tiny = new byte[] {0x00, 0x00, 0x0A, 0x20, 0x00, 0x00, 0x00}; // 32 elements, log_vec=10
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(tiny)));
+ });
+ assertThat(t).as("header-only page must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidOffsetArray() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_vectors = ceil(32/16) = 2, so offset array is 8 bytes. Truncate to chop the 2nd offset.
+ byte[] truncated = new byte[7 + 4]; // header + first offset only
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ });
+ assertThat(t).as("truncated offset array must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidVectorData() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // chop the last 20 bytes — guaranteed to land in the middle of the second vector
+ byte[] truncated = new byte[page.length - 20];
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ // initFromPage should still succeed (truncation is inside the vectors section,
+ // which initFromPage just slices without parsing). The failure surfaces on decode.
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ Throwable t = catchAny(() -> {
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ });
+ assertThat(t).as("truncated vector data must raise on read").isNotNull();
+ }
+
+ @Test
+ public void rejectsCorruptedOffsetPointingPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // Offset array starts at byte 7. Overwrite the first offset (uint32 LE) with a huge value.
+ page[7] = (byte) 0xFF;
+ page[8] = (byte) 0xFF;
+ page[9] = (byte) 0xFF;
+ page[10] = (byte) 0x7F;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ Throwable t = catchAny(() -> reader.readDouble());
+ assertThat(t).as("corrupted offset must raise on decode").isNotNull();
+ }
+
+ // ---------------------------------------------------------------------------
+ // skip() / read() bounds
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsSkipPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(33));
+ }
+
+ @Test
+ public void rejectsNegativeSkip() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(-1));
+ }
+
+ @Test
+ public void rejectsReadPastEnd() throws Exception {
+ byte[] page = validDoublePage(8, 8);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(8, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 8; i++) reader.readDouble();
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exhausted"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Utility
+ // ---------------------------------------------------------------------------
+
+ @FunctionalInterface
+ private interface ThrowingRunnable {
+ void run() throws Throwable;
+ }
+
+ /** Catch any Throwable (including low-level RuntimeExceptions / Errors). */
+ private static Throwable catchAny(ThrowingRunnable r) {
Review Comment:
Catching Throwable lets OOM and assertion failures count as success. This test can pass without proving clean rejection.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
Review Comment:
Does it need to be public or just default visibility?
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isIntrinsicFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure. */
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isIntrinsicDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ // Index positions within an (exponent, factor) pair array.
+ private static final int E = 0;
+ private static final int F = 1;
+
+ // All valid (exponent, factor) pairs for the full search, precomputed in nested-loop order
+ // (e ascending, then f ascending) so the tie-break and early-exit behave exactly like an inline
+ // double loop. Reused across every vector to avoid per-call allocation.
+ private static final int[][] ALL_VALID_FLOAT_PAIRS = buildAllPairs(FLOAT_MAX_EXPONENT);
+ private static final int[][] ALL_VALID_DOUBLE_PAIRS = buildAllPairs(DOUBLE_MAX_EXPONENT);
+
+ private static int[][] buildAllPairs(int maxExponent) {
+ int count = (maxExponent + 1) * (maxExponent + 2) / 2;
+ int[][] pairs = new int[count][];
+ int idx = 0;
+ for (int e = 0; e <= maxExponent; e++) {
+ for (int f = 0; f <= e; f++) {
+ pairs[idx++] = new int[] {e, f};
+ }
+ }
+ return pairs;
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ return pickBestFloat(values, offset, length, ALL_VALID_FLOAT_PAIRS);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ return pickBestFloat(values, offset, length, presets);
+ }
+
+ /**
+ * Scores each (exponent, factor) pair by estimated compressed size and returns the best.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the signed range (max - min) of
+ * non-exception encoded values after frame-of-reference subtraction, matching the writer's FOR
+ * packing. This produces better compression ratios than minimizing exception count alone. Ties in
+ * size are broken toward the higher exponent, then the higher factor. The first pair (in {@code
+ * pairs} order) that encodes every value into a single FOR value with zero exceptions wins
+ * outright. When no pair yields any non-exception values, {@code pairs[0]} is returned.
+ */
+ private static EncodingParams pickBestFloat(float[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
Review Comment:
I'm not sure if this is a performance concern. But `isFloatException(value, e, f)` above has already called `encodeFloat(value, e, f)` internally. Should we consider a method like `tryEncodeFloat` to return a two-state value (Optional?): either good encoded value or indicate it is an exception?
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpValuesEndToEndTest.java:
##########
@@ -0,0 +1,2208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+/**
+ * End-to-end tests for ALP encoding and decoding pipeline.
+ */
+public class AlpValuesEndToEndTest {
+
+ private static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ // ========== Helper Methods ==========
+
+ /**
+ * Round-trip with STRICT raw-bit comparison for every value, including NaN payloads (the regular
+ * roundTrip helpers only assert isNaN, which would hide a NaN-payload-preservation bug). ALP must
+ * be losslessly bit-exact for every possible IEEE-754 bit pattern.
+ */
+ private void roundTripDoubleStrict(double[] values) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(512, values.length * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ long exp = Double.doubleToRawLongBits(values[i]);
+ long act = Double.doubleToRawLongBits(reader.readDouble());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Long.toHexString(exp)
+ + " actualBits=0x" + Long.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ private void roundTripFloatStrict(float[] values) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(256, values.length * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ int exp = Float.floatToRawIntBits(values[i]);
+ int act = Float.floatToRawIntBits(reader.readFloat());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Integer.toHexString(exp)
+ + " actualBits=0x" + Integer.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Full-bit-space fuzz (lossless invariant) ==========
+
+ @Test
+ public void testDoubleFullBitSpaceFuzz() throws Exception {
+ // Sweep the entire double bit space: random raw longs -> double. Covers all subnormals, every
+ // NaN payload, +/-0, +/-Inf, extreme exponents, chaotic mixed magnitudes within a vector.
+ Random rng = new Random(0x9E3779B97F4A7C15L);
+ for (int v = 0; v < 300; v++) { // ~300k values
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Double.longBitsToDouble(rng.nextLong());
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testFloatFullBitSpaceFuzz() throws Exception {
+ Random rng = new Random(0x9E3779B9L);
+ for (int v = 0; v < 300; v++) {
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Float.intBitsToFloat(rng.nextInt());
+ }
+ roundTripFloatStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleMixedFuzz() throws Exception {
+ // Mix ALP-friendly decimal-ish values with random raw bits, so FOR encoding AND the exception
+ // path are both exercised on chaotic data within the same vector.
+ Random rng = new Random(0xD1CE5EEDL);
+ for (int v = 0; v < 300; v++) {
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ if (rng.nextInt(4) == 0) {
+ values[i] = Double.longBitsToDouble(rng.nextLong()); // ~25% chaotic (mostly exceptions)
+ } else {
+ values[i] = Math.round(rng.nextDouble() * 1_000_000.0) / 100.0; // 2-decimal, ALP-friendly
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleDistributionShiftWithinRowGroup() throws Exception {
+ // First vectors are clean 2-decimal data (the sampler builds its preset (e,f) cache from these),
+ // then later vectors switch to high-precision / very different magnitude that the cached presets
+ // fit poorly. Once presets are cached the writer only tries those, so ALP must still stay lossless
+ // (the exception path catches every preset mismatch).
+ Random rng = new Random(7);
+ int totalVectors = 40; // well past SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP so presets are in use
+ double[] values = new double[totalVectors * 1024];
+ for (int i = 0; i < values.length; i++) {
+ int vec = i / 1024;
+ if (vec < 12) {
+ values[i] = Math.round(rng.nextDouble() * 10000.0) / 100.0; // clean cents
+ } else {
+ values[i] = rng.nextDouble() * 1e12 + rng.nextDouble(); // high-precision, big magnitude
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+
+ // ========== Reader robustness against malformed input ==========
+
+ private void readAllDoubles(byte[] bytes, int valueCount) throws Exception {
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(valueCount, ByteBufferInputStream.wrap(ByteBuffer.wrap(bytes)));
+ for (int i = 0; i < valueCount; i++) {
+ reader.readDouble();
+ }
+ }
+
+ @Test
+ @Timeout(value = 30, unit = TimeUnit.SECONDS)
+ public void testReaderRejectsCorruptInputCleanly() throws Exception {
+ // Build a valid ALP double page, then feed corrupted variants. The reader must fail cleanly
+ // (a catchable exception) and never hang, OOM, or read out of bounds silently.
+ double[] values = new double[2048];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (i % 100) / 100.0;
+ }
+ AlpValuesWriter.DoubleAlpValuesWriter writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ 65536, 65536, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ byte[] valid = writer.getBytes().toByteArray();
+ writer.reset();
+ writer.close();
+
+ // Sanity: the valid page reads fine.
+ readAllDoubles(valid, values.length);
+
+ // Corruptions that must each fail cleanly (never crash/hang/OOB):
+ java.util.List<byte[]> corrupt = new java.util.ArrayList<>();
+ corrupt.add(java.util.Arrays.copyOf(valid, valid.length / 2)); // truncated to half
+ corrupt.add(java.util.Arrays.copyOf(valid, 3)); // truncated to a stub
+ corrupt.add(new byte[0]); // empty
+ for (int pos : new int[] {0, 1, 2, 5, 7, 11, 20, valid.length - 1}) {
+ byte[] c = valid.clone();
+ c[pos] = (byte) ~c[pos]; // flip a header/body byte
+ corrupt.add(c);
+ }
+ for (byte[] c : corrupt) {
+ try {
+ readAllDoubles(c, values.length);
+ // Some single-byte flips may still decode to (wrong but in-bounds) values without throwing;
+ // that is acceptable here. What matters is no crash/hang/OOB, which we reached this line.
+ } catch (OutOfMemoryError oom) {
+ fail("Malformed input caused an OutOfMemoryError (allocation bomb) - a corrupt size/count "
+ + "must not drive an unbounded allocation");
+ } catch (Throwable t) {
+ // Any ordinary catchable exception (EOFException, ParquetDecodingException, IndexOutOfBounds,
+ // BufferUnderflow, NegativeArraySize, ...) is a clean failure: no JVM crash, no OOB, no hang.
+ }
+ }
+
+ // Claiming far more elements than the data supports must be rejected without an OOM allocation.
+ try {
+ readAllDoubles(valid, Integer.MAX_VALUE / 2);
Review Comment:
This changes valuesCount, not the ALP header num_elements. It does not exercise the allocation or overflow path.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
+ Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[PACK_GROUP_SIZE];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Sampling phase first (full search + collect evenly-spaced samples, then build the preset
+ // cache once enough are gathered); after the cache is built, later vectors take the else branch.
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets == null) {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && sampledParams.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ sampledParams.add(new int[] {params.exponent, params.factor});
+ }
+ if (sampledParams.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
Review Comment:
`vectorsProcessed` resets for every page. With the default page size, we only collect about two samples, but need eight to build the cache. So `cachedPresets` is never used in the normal path.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java:
##########
@@ -147,7 +148,13 @@ private ValuesWriter getInt96ValuesWriter(ColumnDescriptor path) {
private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) {
final ValuesWriter fallbackWriter;
- if (this.parquetProperties.isByteStreamSplitEnabled(path)) {
+ if (this.parquetProperties.isAlpEnabled(path)) {
Review Comment:
+1 on `keep ALP registered in both V1 and V2` to avoid confusion and it is aligned with the change of `isByteStreamSplitEnabled`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ if (n < 0 || pageValueIndex + n > totalCount) {
Review Comment:
This addition can overflow. A large skip can pass the check and make pageValueIndex negative.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
+
+ private AlpConstants() {
+ // Utility class
+ }
+
+ // Page header fields
+ public static final int ALP_COMPRESSION_MODE = 0;
+ public static final int ALP_INTEGER_ENCODING_FOR = 0;
+ public static final int ALP_HEADER_SIZE = 7;
+
+ public static final int DEFAULT_VECTOR_SIZE = 1024;
+ public static final int DEFAULT_VECTOR_SIZE_LOG = 10;
+
+ // BytePacker packs/unpacks 8 values at a time (pack8Values/unpack8Values).
+ static final int PACK_GROUP_SIZE = 8;
+
+ // Capped at 15 (vectorSize=32768) because num_exceptions is uint16,
+ // so vectorSize must not exceed 65535 to avoid overflow when all values are exceptions.
+ static final int MAX_LOG_VECTOR_SIZE = 15;
+ static final int MIN_LOG_VECTOR_SIZE = 3;
+
+ static final int FLOAT_MAX_EXPONENT = 10;
+ static final int DOUBLE_MAX_EXPONENT = 18;
+
+ // Sampler constants matching C++ AlpConstants.
+ // Sample SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP vectors evenly distributed across a rowgroup
+ // of SAMPLER_ROWGROUP_SIZE values, then lock in top MAX_PRESET_COMBINATIONS combos.
+ static final int SAMPLER_ROWGROUP_SIZE = 122_880;
Review Comment:
Unless shared by both encoder and decoder, variables like these are better placed in the file that directly uses them.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
Review Comment:
This line is not aligned on my editor.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
Review Comment:
Please do not use wildcard import. Same to other files.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
Review Comment:
nit: rename it to `AlpCodec` or `AlpUtil`
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
Review Comment:
Why the writer implementations are nested static class but the readers are in the separate files? Should we make them consistent?
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +637,74 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Enable or disable ALP encoding for FLOAT and DOUBLE columns.
+ *
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(boolean enable) {
+ this.alpEnabled.withDefaultValue(enable);
+ return this;
+ }
+
+ /**
+ * Enable or disable ALP encoding for the specified column.
+ *
+ * @param columnPath the path of the column (dot-string)
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(String columnPath, boolean enable) {
+ this.alpEnabled.withValue(columnPath, enable);
+ return this;
+ }
+
+ /**
+ * Set the ALP vector size (number of values per encoded vector) for FLOAT and DOUBLE columns.
+ * Must be a power of 2 in the range supported by {@link AlpConstants}.
+ *
+ * @param vectorSize the vector size
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpVectorSize(int vectorSize) {
Review Comment:
Why not directly use `withAlp(AlpConfig)`? It looks a little bit over-complicated with the current API, especially `buildAlp()`.
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1408 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {"alp_float_spotify1.parquet", "alp_float_arade.parquet"};
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
Review Comment:
This does not follow other interop tests in the repo to download it from parquet-testing repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-5686025101
@RussellSpitzer with #3791 merged, I've merged master in and worked through what was left of your nits:
- **Thrift**: the inlined `parquet.thrift` is the upstream 2.14.0 file, so the local addition is gone.
- **Skips**: the `testEnumEquivalence` ALP skip is gone again (7b48dd189), after the merge reintroduced it.
- **Javadocs**: I grepped again for anything describing ALP as not yet upstream. The only thing left was a `@see` to parquet-testing#100, which has since been closed in favour of #119, so I dropped it.
- **`withoutAlp(column)`**: in a8c584f64, as before.
One question from last time is still open: `TestInterOpReadAlp` skips when pyarrow reports "Unknown encoding type". Happy to pull it if you'd rather it fail loudly.
Your approval is against e660c6cec, so another look whenever you get a chance would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4019578633
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1408 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {"alp_float_spotify1.parquet", "alp_float_arade.parquet"};
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
Review Comment:
Done in bd206953a. #100 was superseded by apache/parquet-testing#119, which has merged, so `TestInterOpReadAlp` now fetches `alp_extended.zstd.parquet` through `InterOpTester` at a pinned changeset, like the other interop tests. Every ALP column is checked bit for bit against its PLAIN counterpart across all 9032 rows, with spot checks on the NaN payloads, infinities, -0.0, the smallest subnormal and the ±8e18 rows. The local-directory tests and the fixture generators that went with them are gone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3942046622
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
Review Comment:
Thanks for catching this, it's a good find. You're right that numElements is only bounded by valuesCount, and since a forged file controls that value it can go all the way up to Integer.MAX_VALUE. Two separate things went wrong from there. The `+ vectorSize - 1` overflowed into a negative count, and even when it didn't overflow, a large count drove a big allocation before anything had been validated.
While fixing it I found the allocation is genuinely reachable rather than theoretical. MultiBufferInputStream.slice() calls ByteBuffer.allocate(length) before it checks for EOF, so a small forged page really can ask for hundreds of megabytes.
This is fixed in 338fb99c6. numVectors is computed in long now and checked against the bytes actually present before anything gets allocated. Since every vector needs a 4 byte offset entry plus at least its ALP and FOR headers, that gives a firm ceiling on how many vectors the remaining bytes could possibly describe, and anything above it throws a ParquetDecodingException that says how far short the page is.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
Review Comment:
You're right on all three counts, thank you. Fixed in 338fb99c6.
initFromPage now validates the whole offset array before any vector is decoded. The first offset has to equal the offset array size, offsets have to increase by at least one vector's worth of fixed headers, and each one has to start early enough to leave room for those headers before the body ends.
The observation about decode ignoring the next offset was the more important half of this, since that's the case that quietly returns wrong values instead of failing. decodeVector now takes each vector's end position from the following offset, or from the end of the body for the last vector, and rejects a packed body or exception block that runs past it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
+ ByteBuffer headerBuf = stream.slice(ALP_HEADER_SIZE).order(ByteOrder.LITTLE_ENDIAN);
+ int compressionMode = headerBuf.get() & 0xFF;
+ int integerEncoding = headerBuf.get() & 0xFF;
+ int logVectorSize = headerBuf.get() & 0xFF;
+ int numElements = headerBuf.getInt();
+
+ if (compressionMode != ALP_COMPRESSION_MODE) {
+ throw new ParquetDecodingException("Unsupported ALP compression mode: " + compressionMode);
+ }
+ if (integerEncoding != ALP_INTEGER_ENCODING_FOR) {
+ throw new ParquetDecodingException("Unsupported ALP integer encoding: " + integerEncoding);
+ }
+ if (logVectorSize < MIN_LOG_VECTOR_SIZE || logVectorSize > MAX_LOG_VECTOR_SIZE) {
+ throw new ParquetDecodingException("Invalid ALP log vector size: " + logVectorSize + ", must be between "
+ + MIN_LOG_VECTOR_SIZE + " and " + MAX_LOG_VECTOR_SIZE);
+ }
+ if (numElements < 0) {
+ throw new ParquetDecodingException("Invalid ALP element count: " + numElements);
+ }
+ // ALP's num_elements is the count of non-null values that went through encoding;
+ // valuesCount is the page row count, which is larger when the column has nulls.
+ // The two are equal only for required (non-null) columns.
+ if (numElements > valuesCount) {
+ throw new ParquetDecodingException(
+ "ALP header element count " + numElements + " exceeds page valuesCount " + valuesCount);
+ }
+
+ this.vectorSize = 1 << logVectorSize;
+ this.totalCount = numElements;
+ this.numVectors = (numElements + vectorSize - 1) / vectorSize;
+ this.pageValueIndex = 0;
+ this.currentVectorNumber = -1;
+
+ this.offsetArraySize = numVectors * Integer.BYTES;
+ ByteBuffer offsetBuf = stream.slice(offsetArraySize).order(ByteOrder.LITTLE_ENDIAN);
+ this.vectorOffsets = new int[numVectors];
+ for (int v = 0; v < numVectors; v++) {
+ vectorOffsets[v] = offsetBuf.getInt();
+ }
+
+ // Slice remaining bytes into a 0-based view so decodeVector can use
+ // absolute get methods (vectorsData.get(pos)) directly.
+ int remainingBytes = (int) stream.available();
+ ByteBuffer rawSlice = stream.slice(remainingBytes);
+ this.vectorsData = rawSlice.slice().order(ByteOrder.LITTLE_ENDIAN);
+
+ allocateDecodedBuffer(vectorSize);
+ this.excPositionsBuffer = new int[vectorSize];
+ }
+
+ protected int getVectorLength(int vectorNumber) {
+ if (vectorNumber < numVectors - 1) {
+ return vectorSize;
+ }
+ // Last vector may be partial
+ int lastVectorLen = totalCount % vectorSize;
+ return lastVectorLen == 0 ? vectorSize : lastVectorLen;
+ }
+
+ // Offsets in the page are relative to the compression body (after header),
+ // but vectorsData starts after the offset array, so adjust.
+ protected int getVectorDataPosition(int vectorNumber) {
+ return vectorOffsets[vectorNumber] - offsetArraySize;
+ }
+
+ @Override
+ public void skip() {
+ skip(1);
+ }
+
+ @Override
+ public void skip(int n) {
+ if (n < 0 || pageValueIndex + n > totalCount) {
Review Comment:
Good catch, thank you. Fixed in 338fb99c6 by comparing against the remaining count with `n > totalCount - pageValueIndex` rather than adding to pageValueIndex, so a large n can no longer overflow past the check and leave the reader at a negative index. I added testSkipRejectsOverflowingCount to cover it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
Review Comment:
Thank you for working through the arithmetic on this one, it turned out to be the most valuable comment in the review. I checked it and the numbers come out exactly as you described. At the default 20k row page limit with a 1024 vector size a page holds roughly 19 vectors, and rowgroupSampleJump works out to 15, so each page contributes about two samples. Because reset() cleared sampledParams and vectorsProcessed at every page boundary, the count never got near SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP, buildPresetCache() never ran, and findBest*ParamsWithPresets was effectively dead code. Every vector was paying for a full parameter search.
Fixed in eb0e5941d. The sampler is meant to operate over a rowgroup rather than a page, so the sampling state now survives reset() and samples accumulate across pages the way the design intended. I added testPresetCacheBuildsAcrossPageBoundaries, which fails without the change.
One consequence I want to flag, since it isn't obvious. Now that the preset path is genuinely reachable, encoded output can shift slightly, because the search narrows to the top MAX_PRESET_COMBINATIONS pairs instead of trying everything. That is the intended ALP sampling behavior and it matches the C++ implementation. Round tripping is still exact, since the chosen pair is recorded in each vector's header and exceptions cover anything it cannot represent.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector contains interleaved:
+ * AlpInfo(4B) + ForInfo(5B/9B) + PackedValues + ExceptionPositions + ExceptionValues
+ */
+public abstract class AlpValuesWriter extends ValuesWriter {
+
+ protected final int initialCapacity;
+ protected final int pageSize;
+ protected final ByteBufferAllocator allocator;
+ protected final int vectorSize;
+ protected final int logVectorSize;
+
+ AlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ AlpConstants.validateVectorSize(vectorSize);
+ this.initialCapacity = initialCapacity;
+ this.pageSize = pageSize;
+ this.allocator = allocator;
+ this.vectorSize = vectorSize;
+ this.logVectorSize = Integer.numberOfTrailingZeros(vectorSize);
+ }
+
+ @Override
+ public Encoding getEncoding() {
+ return Encoding.ALP;
+ }
+
+ /** Float writer. Buffers one vector at a time, encodes and flushes when full. */
+ public static class FloatAlpValuesWriter extends AlpValuesWriter {
+ private final float[] vectorBuffer;
+ private int bufferCount;
+ private int totalCount;
+ private CapacityByteArrayOutputStream encodedVectors;
+ private final List<Integer> vectorByteSizes;
+
+ // Preset caching: collect evenly-spaced sample vectors across the rowgroup,
+ // then build presets using estimated compressed size (matching C++ AlpSampler).
+ private int vectorsProcessed;
+ private int[][] cachedPresets;
+ // Winning (exponent, factor) pairs from sampled vectors, tallied later into the preset cache.
+ private final List<int[]> sampledParams;
+ private final int rowgroupSampleJump;
+
+ // Reusable per-vector buffers
+ private final int[] encodedBuffer;
+ private final short[] excPosBuffer;
+ private final float[] excValBuffer;
+ private final byte[] metadataBuf;
+ private final byte[] packBuf;
+ private final int[] packPadBuf;
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator) {
+ this(initialCapacity, pageSize, allocator, DEFAULT_VECTOR_SIZE);
+ }
+
+ public FloatAlpValuesWriter(int initialCapacity, int pageSize, ByteBufferAllocator allocator, int vectorSize) {
+ super(initialCapacity, pageSize, allocator, vectorSize);
+ this.vectorBuffer = new float[vectorSize];
+ this.bufferCount = 0;
+ this.totalCount = 0;
+ this.encodedVectors = new CapacityByteArrayOutputStream(initialCapacity, pageSize, allocator);
+ this.vectorByteSizes = new ArrayList<>();
+ this.vectorsProcessed = 0;
+ this.cachedPresets = null;
+ this.sampledParams = new ArrayList<>();
+ // Space samples evenly: one sample every jump vectors across the rowgroup.
+ // Math.max(1, ...) guards against very small rowgroups or large vector sizes.
+ this.rowgroupSampleJump =
+ Math.max(1, SAMPLER_ROWGROUP_SIZE / SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP / vectorSize);
+ // Pre-allocate reusable buffers
+ this.encodedBuffer = new int[vectorSize];
+ this.excPosBuffer = new short[vectorSize];
+ this.excValBuffer = new float[vectorSize];
+ this.metadataBuf = new byte[Math.max(ALP_INFO_SIZE, FLOAT_FOR_INFO_SIZE)];
+ this.packBuf = new byte[Integer.SIZE]; // max bit width for int
+ this.packPadBuf = new int[PACK_GROUP_SIZE];
+ }
+
+ @Override
+ public void writeFloat(float v) {
+ vectorBuffer[bufferCount++] = v;
+ totalCount++;
+ if (bufferCount == vectorSize) {
+ encodeAndFlushVector(bufferCount);
+ bufferCount = 0;
+ }
+ }
+
+ private void encodeAndFlushVector(int vectorLen) {
+ // Sampling phase first (full search + collect evenly-spaced samples, then build the preset
+ // cache once enough are gathered); after the cache is built, later vectors take the else branch.
+ AlpEncoderDecoder.EncodingParams params;
+ if (cachedPresets == null) {
+ params = AlpEncoderDecoder.findBestFloatParams(vectorBuffer, 0, vectorLen);
+ // Collect one sample every rowgroupSampleJump vectors so that samples are
+ // evenly distributed across the rowgroup (matching C++ AlpSampler spacing).
+ if (vectorsProcessed % rowgroupSampleJump == 0
+ && sampledParams.size() < SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ sampledParams.add(new int[] {params.exponent, params.factor});
+ }
+ if (sampledParams.size() >= SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP) {
+ buildPresetCache();
+ }
+ } else {
+ params = AlpEncoderDecoder.findBestFloatParamsWithPresets(vectorBuffer, 0, vectorLen, cachedPresets);
Review Comment:
This is the same underlying problem as your comment on the sampling cadence, and it's fixed in eb0e5941d. I've written up the details in that thread rather than repeating them here. Thanks for flagging it from both angles, it made the problem much easier to pin down.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpValuesEndToEndTest.java:
##########
@@ -0,0 +1,2208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.Random;
+import java.util.concurrent.TimeUnit;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.Packer;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Timeout;
+
+/**
+ * End-to-end tests for ALP encoding and decoding pipeline.
+ */
+public class AlpValuesEndToEndTest {
+
+ private static final int DEFAULT_VECTOR_SIZE = AlpConstants.DEFAULT_VECTOR_SIZE;
+
+ // ========== Helper Methods ==========
+
+ /**
+ * Round-trip with STRICT raw-bit comparison for every value, including NaN payloads (the regular
+ * roundTrip helpers only assert isNaN, which would hide a NaN-payload-preservation bug). ALP must
+ * be losslessly bit-exact for every possible IEEE-754 bit pattern.
+ */
+ private void roundTripDoubleStrict(double[] values) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(512, values.length * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ long exp = Double.doubleToRawLongBits(values[i]);
+ long act = Double.doubleToRawLongBits(reader.readDouble());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Long.toHexString(exp)
+ + " actualBits=0x" + Long.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ private void roundTripFloatStrict(float[] values) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int capacity = Math.max(256, values.length * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(
+ capacity, capacity, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (float v : values) {
+ writer.writeFloat(v);
+ }
+ BytesInput input = writer.getBytes();
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(values.length, ByteBufferInputStream.wrap(input.toByteBuffer()));
+ for (int i = 0; i < values.length; i++) {
+ int exp = Float.floatToRawIntBits(values[i]);
+ int act = Float.floatToRawIntBits(reader.readFloat());
+ assertThat(act)
+ .as("Raw-bit mismatch at index " + i + " expectedBits=0x" + Integer.toHexString(exp)
+ + " actualBits=0x" + Integer.toHexString(act))
+ .isEqualTo(exp);
+ }
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ // ========== Full-bit-space fuzz (lossless invariant) ==========
+
+ @Test
+ public void testDoubleFullBitSpaceFuzz() throws Exception {
+ // Sweep the entire double bit space: random raw longs -> double. Covers all subnormals, every
+ // NaN payload, +/-0, +/-Inf, extreme exponents, chaotic mixed magnitudes within a vector.
+ Random rng = new Random(0x9E3779B97F4A7C15L);
+ for (int v = 0; v < 300; v++) { // ~300k values
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Double.longBitsToDouble(rng.nextLong());
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testFloatFullBitSpaceFuzz() throws Exception {
+ Random rng = new Random(0x9E3779B9L);
+ for (int v = 0; v < 300; v++) {
+ float[] values = new float[1024];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = Float.intBitsToFloat(rng.nextInt());
+ }
+ roundTripFloatStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleMixedFuzz() throws Exception {
+ // Mix ALP-friendly decimal-ish values with random raw bits, so FOR encoding AND the exception
+ // path are both exercised on chaotic data within the same vector.
+ Random rng = new Random(0xD1CE5EEDL);
+ for (int v = 0; v < 300; v++) {
+ double[] values = new double[1024];
+ for (int i = 0; i < values.length; i++) {
+ if (rng.nextInt(4) == 0) {
+ values[i] = Double.longBitsToDouble(rng.nextLong()); // ~25% chaotic (mostly exceptions)
+ } else {
+ values[i] = Math.round(rng.nextDouble() * 1_000_000.0) / 100.0; // 2-decimal, ALP-friendly
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+ }
+
+ @Test
+ public void testDoubleDistributionShiftWithinRowGroup() throws Exception {
+ // First vectors are clean 2-decimal data (the sampler builds its preset (e,f) cache from these),
+ // then later vectors switch to high-precision / very different magnitude that the cached presets
+ // fit poorly. Once presets are cached the writer only tries those, so ALP must still stay lossless
+ // (the exception path catches every preset mismatch).
+ Random rng = new Random(7);
+ int totalVectors = 40; // well past SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP so presets are in use
+ double[] values = new double[totalVectors * 1024];
+ for (int i = 0; i < values.length; i++) {
+ int vec = i / 1024;
+ if (vec < 12) {
+ values[i] = Math.round(rng.nextDouble() * 10000.0) / 100.0; // clean cents
+ } else {
+ values[i] = rng.nextDouble() * 1e12 + rng.nextDouble(); // high-precision, big magnitude
+ }
+ }
+ roundTripDoubleStrict(values);
+ }
+
+ // ========== Reader robustness against malformed input ==========
+
+ private void readAllDoubles(byte[] bytes, int valueCount) throws Exception {
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(valueCount, ByteBufferInputStream.wrap(ByteBuffer.wrap(bytes)));
+ for (int i = 0; i < valueCount; i++) {
+ reader.readDouble();
+ }
+ }
+
+ @Test
+ @Timeout(value = 30, unit = TimeUnit.SECONDS)
+ public void testReaderRejectsCorruptInputCleanly() throws Exception {
+ // Build a valid ALP double page, then feed corrupted variants. The reader must fail cleanly
+ // (a catchable exception) and never hang, OOM, or read out of bounds silently.
+ double[] values = new double[2048];
+ for (int i = 0; i < values.length; i++) {
+ values[i] = (i % 100) / 100.0;
+ }
+ AlpValuesWriter.DoubleAlpValuesWriter writer = new AlpValuesWriter.DoubleAlpValuesWriter(
+ 65536, 65536, new DirectByteBufferAllocator(), DEFAULT_VECTOR_SIZE);
+ for (double v : values) {
+ writer.writeDouble(v);
+ }
+ byte[] valid = writer.getBytes().toByteArray();
+ writer.reset();
+ writer.close();
+
+ // Sanity: the valid page reads fine.
+ readAllDoubles(valid, values.length);
+
+ // Corruptions that must each fail cleanly (never crash/hang/OOB):
+ java.util.List<byte[]> corrupt = new java.util.ArrayList<>();
+ corrupt.add(java.util.Arrays.copyOf(valid, valid.length / 2)); // truncated to half
+ corrupt.add(java.util.Arrays.copyOf(valid, 3)); // truncated to a stub
+ corrupt.add(new byte[0]); // empty
+ for (int pos : new int[] {0, 1, 2, 5, 7, 11, 20, valid.length - 1}) {
+ byte[] c = valid.clone();
+ c[pos] = (byte) ~c[pos]; // flip a header/body byte
+ corrupt.add(c);
+ }
+ for (byte[] c : corrupt) {
+ try {
+ readAllDoubles(c, values.length);
+ // Some single-byte flips may still decode to (wrong but in-bounds) values without throwing;
+ // that is acceptable here. What matters is no crash/hang/OOB, which we reached this line.
+ } catch (OutOfMemoryError oom) {
+ fail("Malformed input caused an OutOfMemoryError (allocation bomb) - a corrupt size/count "
+ + "must not drive an unbounded allocation");
+ } catch (Throwable t) {
+ // Any ordinary catchable exception (EOFException, ParquetDecodingException, IndexOutOfBounds,
+ // BufferUnderflow, NegativeArraySize, ...) is a clean failure: no JVM crash, no OOB, no hang.
+ }
+ }
+
+ // Claiming far more elements than the data supports must be rejected without an OOM allocation.
+ try {
+ readAllDoubles(valid, Integer.MAX_VALUE / 2);
Review Comment:
You're right, and thank you for reading the tests as carefully as the production code. That test only inflated the page's valuesCount, so the `numElements <= valuesCount` check passed trivially and the allocation path was never touched.
I replaced it in 338fb99c6 with testForgedElementCountIsRejectedWithoutHugeAllocation, which overwrites the num_elements field in the header directly and passes Integer.MAX_VALUE as valuesCount so that the allocation bound is what has to reject it. It covers both the counts that are simply too large for the page and the two that overflow the vector count arithmetic.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/alp/AlpAdversarialTest.java:
##########
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.fail;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.DirectByteBufferAllocator;
+import org.apache.parquet.io.ParquetDecodingException;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Adversarial tests for ALP readers: feed malformed page bytes and assert the reader
+ * fails cleanly rather than crashing, producing silent garbage, or hanging.
+ *
+ * <p>"Fails cleanly" means raising a meaningful exception — preferably
+ * {@link ParquetDecodingException}, but at minimum a typed exception (not a JVM-level
+ * crash, infinite loop, or wrong answer). The tests cover both:
+ * <ul>
+ * <li>Already-validated cases — the reader explicitly rejects these with a
+ * ParquetDecodingException carrying an explanatory message. These tests pin
+ * the validation behavior in place.
+ * <li>Currently-unvalidated cases (truncation, corrupted offsets) — the reader
+ * relies on the underlying ByteBuffer to surface IndexOutOfBoundsException or
+ * BufferUnderflowException. These tests assert that some Throwable is raised
+ * so the failure mode stays "loud" even if the explicit message is missing.
+ * </ul>
+ */
+public class AlpAdversarialTest {
+
+ // ---------------------------------------------------------------------------
+ // Helpers: build a known-good encoded page, then mutate copies of it
+ // ---------------------------------------------------------------------------
+
+ /** Build a valid ALP-encoded double page with N clean values. */
+ private static byte[] validDoublePage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.DoubleAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(512, valueCount * 16);
+ writer = new AlpValuesWriter.DoubleAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ // 2-decimal values — the ALP sweet spot, ensures no exceptions
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeDouble((i % 1000) / 100.0);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Build a valid ALP-encoded float page with N clean values. */
+ private static byte[] validFloatPage(int valueCount, int vectorSize) throws Exception {
+ AlpValuesWriter.FloatAlpValuesWriter writer = null;
+ try {
+ int cap = Math.max(256, valueCount * 8);
+ writer = new AlpValuesWriter.FloatAlpValuesWriter(cap, cap, new DirectByteBufferAllocator(), vectorSize);
+ for (int i = 0; i < valueCount; i++) {
+ writer.writeFloat((i % 1000) / 100.0f);
+ }
+ BytesInput bi = writer.getBytes();
+ ByteBuffer bb = bi.toByteBuffer();
+ byte[] out = new byte[bb.remaining()];
+ bb.duplicate().get(out);
+ return out;
+ } finally {
+ if (writer != null) {
+ writer.reset();
+ writer.close();
+ }
+ }
+ }
+
+ /** Sanity baseline: the known-good page actually decodes cleanly. */
+ @Test
+ public void sanityBaselineDecodesClean() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Header-level validation (already-validated paths)
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsBadCompressionMode() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[0] = (byte) 0x99; // mode is at byte 0
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("compression"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBadIntegerEncoding() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[1] = (byte) 0x99; // integer_encoding is at byte 1
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("integer encoding"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooLarge() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 99; // log_vector_size at byte 2
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("vector size"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsLogVectorSizeTooSmall() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ page[2] = (byte) 2; // below MIN_LOG_VECTOR_SIZE=3
+ assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ }
+
+ @Test
+ public void rejectsNegativeNumElements() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements is int32 LE at bytes 3..6 — write -1
+ page[3] = (byte) 0xFF;
+ page[4] = (byte) 0xFF;
+ page[5] = (byte) 0xFF;
+ page[6] = (byte) 0xFF;
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("element count"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsNumElementsGreaterThanValuesCount() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_elements stays 32; pass valuesCount=10 (smaller than encoded count)
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, () -> {
+ new AlpValuesReaderForDouble().initFromPage(10, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ });
+ assertThat(ex.getMessage().toLowerCase().contains("exceeds"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Vector-level validation (already-validated paths, surface lazily on decode)
+ // ---------------------------------------------------------------------------
+
+ /** Helper: find the byte position where the first vector's metadata starts. */
+ private static int firstVectorOffset(byte[] page) {
+ // header (7) + first 4 bytes of offset array = the offset value itself
+ int firstVectorOff =
+ ByteBuffer.wrap(page, 7, 4).order(ByteOrder.LITTLE_ENDIAN).getInt();
+ // offsets are measured from the start of the compression body (after the 7B header)
+ return 7 + firstVectorOff;
+ }
+
+ @Test
+ public void rejectsExponentTooHighDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99; // exponent byte
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsExponentTooHighFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("exponent"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsFactorGreaterThanExponent() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ page[v0] = (byte) 2; // exponent
+ page[v0 + 1] = (byte) 5; // factor > exponent
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("factor"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsTooManyExceptions() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // num_exceptions at v0+2, uint16 LE — set to 9999, way more than vectorLen=16
+ page[v0 + 2] = (byte) (9999 & 0xFF);
+ page[v0 + 3] = (byte) ((9999 >>> 8) & 0xFF);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("numexceptions"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeDouble() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(8) then bitWidth byte at v0+12. 99 > 64.
+ page[v0 + 12] = (byte) 99;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ @Test
+ public void rejectsBitWidthTooLargeFloat() throws Exception {
+ byte[] page = validFloatPage(32, 16);
+ int v0 = firstVectorOffset(page);
+ // Layout: ALP_INFO(4) + frameOfReference(4) then bitWidth byte at v0+8. 99 > 32.
+ page[v0 + 8] = (byte) 99;
+ AlpValuesReaderForFloat reader = new AlpValuesReaderForFloat();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readFloat);
+ assertThat(ex.getMessage().toLowerCase().contains("bitwidth"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Currently-unvalidated paths: truncation and corrupted offsets
+ // These currently fail with low-level Throwables (BufferUnderflowException,
+ // IndexOutOfBoundsException). The tests assert any Throwable is raised so we
+ // notice if a regression silently swallows the corruption.
+ // ---------------------------------------------------------------------------
+
+ /** Page with only the 7-byte header — nothing else. */
+ @Test
+ public void rejectsHeaderOnlyPage() {
+ byte[] tiny = new byte[] {0x00, 0x00, 0x0A, 0x20, 0x00, 0x00, 0x00}; // 32 elements, log_vec=10
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(tiny)));
+ });
+ assertThat(t).as("header-only page must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidOffsetArray() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // num_vectors = ceil(32/16) = 2, so offset array is 8 bytes. Truncate to chop the 2nd offset.
+ byte[] truncated = new byte[7 + 4]; // header + first offset only
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ Throwable t = catchAny(() -> {
+ new AlpValuesReaderForDouble().initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ });
+ assertThat(t).as("truncated offset array must raise").isNotNull();
+ }
+
+ @Test
+ public void rejectsPageTruncatedMidVectorData() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // chop the last 20 bytes — guaranteed to land in the middle of the second vector
+ byte[] truncated = new byte[page.length - 20];
+ System.arraycopy(page, 0, truncated, 0, truncated.length);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ // initFromPage should still succeed (truncation is inside the vectors section,
+ // which initFromPage just slices without parsing). The failure surfaces on decode.
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated)));
+ Throwable t = catchAny(() -> {
+ for (int i = 0; i < 32; i++) reader.readDouble();
+ });
+ assertThat(t).as("truncated vector data must raise on read").isNotNull();
+ }
+
+ @Test
+ public void rejectsCorruptedOffsetPointingPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ // Offset array starts at byte 7. Overwrite the first offset (uint32 LE) with a huge value.
+ page[7] = (byte) 0xFF;
+ page[8] = (byte) 0xFF;
+ page[9] = (byte) 0xFF;
+ page[10] = (byte) 0x7F;
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ Throwable t = catchAny(() -> reader.readDouble());
+ assertThat(t).as("corrupted offset must raise on decode").isNotNull();
+ }
+
+ // ---------------------------------------------------------------------------
+ // skip() / read() bounds
+ // ---------------------------------------------------------------------------
+
+ @Test
+ public void rejectsSkipPastEnd() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(33));
+ }
+
+ @Test
+ public void rejectsNegativeSkip() throws Exception {
+ byte[] page = validDoublePage(32, 16);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(32, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ assertThrows(ParquetDecodingException.class, () -> reader.skip(-1));
+ }
+
+ @Test
+ public void rejectsReadPastEnd() throws Exception {
+ byte[] page = validDoublePage(8, 8);
+ AlpValuesReaderForDouble reader = new AlpValuesReaderForDouble();
+ reader.initFromPage(8, ByteBufferInputStream.wrap(ByteBuffer.wrap(page)));
+ for (int i = 0; i < 8; i++) reader.readDouble();
+ ParquetDecodingException ex = assertThrows(ParquetDecodingException.class, reader::readDouble);
+ assertThat(ex.getMessage().toLowerCase().contains("exhausted"))
+ .as(ex.getMessage())
+ .isTrue();
+ }
+
+ // ---------------------------------------------------------------------------
+ // Utility
+ // ---------------------------------------------------------------------------
+
+ @FunctionalInterface
+ private interface ThrowingRunnable {
+ void run() throws Throwable;
+ }
+
+ /** Catch any Throwable (including low-level RuntimeExceptions / Errors). */
+ private static Throwable catchAny(ThrowingRunnable r) {
Review Comment:
Agreed, and thanks for pointing it out. Catching Throwable meant the test could not really fail, which defeated the point of it.
In 338fb99c6 I replaced catchAny with catchClean, which catches Exception and asserts that the type is one of the expected clean failures while letting Error propagate. An OutOfMemoryError from an allocation bomb or an AssertionError from an inner assertion will now fail the test rather than quietly counting as a successful rejection. I applied the same change to the `catch (Throwable)` in the corruption loop in AlpValuesEndToEndTest.
##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -585,6 +637,74 @@ public Builder withExtendedByteStreamSplitEncoding(boolean enable) {
return this;
}
+ /**
+ * Enable or disable ALP encoding for FLOAT and DOUBLE columns.
+ *
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(boolean enable) {
+ this.alpEnabled.withDefaultValue(enable);
+ return this;
+ }
+
+ /**
+ * Enable or disable ALP encoding for the specified column.
+ *
+ * @param columnPath the path of the column (dot-string)
+ * @param enable whether ALP encoding should be enabled
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpEncoding(String columnPath, boolean enable) {
+ this.alpEnabled.withValue(columnPath, enable);
+ return this;
+ }
+
+ /**
+ * Set the ALP vector size (number of values per encoded vector) for FLOAT and DOUBLE columns.
+ * Must be a power of 2 in the range supported by {@link AlpConstants}.
+ *
+ * @param vectorSize the vector size
+ * @return this builder for method chaining.
+ */
+ public Builder withAlpVectorSize(int vectorSize) {
Review Comment:
You're right that this was more complicated than it needed to be, thanks for pushing on it. The builder was keeping the enabled flag and the vector size as two separate ColumnProperty scaffolds, merging them into an AlpConfig in buildAlp(), and then splitting them apart again in the copy constructor. Nothing actually needed that round trip.
Fixed in 99ae547c5. There is now a single ColumnProperty<AlpConfig>, with withAlp(AlpConfig) and withAlp(columnPath, AlpConfig) as the primary API. I kept withAlpEncoding and withAlpVectorSize as thin read-modify-write conveniences on top, since they match the ergonomics of withByteStreamSplitEncoding and it means no existing caller had to change, including ParquetWriter.Builder. buildAlp and the copy constructor split are both gone.
I also moved the vector size validation into the AlpConfig constructor, which felt like the right home for it since an invalid size should be rejected however the config was built. That had the nice side effect of removing the last reference to AlpConstants from ParquetProperties, which made your visibility comment straightforward to act on.
One thing I want to flag for you. To let the convenience setters modify an existing config rather than replace it, I added getDefaultValue and getValue to ColumnProperty.Builder. That class is package private so it is not public API, but it is a shared file rather than something ALP specific, so please say if you would rather I found another way around it.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
Review Comment:
Good question, and the answer is that it did not need to be public. I checked every reference and ParquetProperties was the only thing outside the package touching it, for DEFAULT_VECTOR_SIZE and validateVectorSize. Once the AlpConfig change above removed both of those, nothing external was left.
The class and all of its members are package private now, in 78c1d182a. It seemed worth doing properly before a release rather than after, since it keeps the ALP internals out of the public API surface permanently.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpConstants.java:
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import org.apache.parquet.Preconditions;
+
+/**
+ * Constants for the ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Based on the paper: "ALP: Adaptive Lossless floating-Point Compression" (SIGMOD 2024)
+ *
+ * @see <a href="https://dl.acm.org/doi/10.1145/3626717">ALP Paper</a>
+ */
+public final class AlpConstants {
+
+ private AlpConstants() {
+ // Utility class
+ }
+
+ // Page header fields
+ public static final int ALP_COMPRESSION_MODE = 0;
+ public static final int ALP_INTEGER_ENCODING_FOR = 0;
+ public static final int ALP_HEADER_SIZE = 7;
+
+ public static final int DEFAULT_VECTOR_SIZE = 1024;
+ public static final int DEFAULT_VECTOR_SIZE_LOG = 10;
+
+ // BytePacker packs/unpacks 8 values at a time (pack8Values/unpack8Values).
+ static final int PACK_GROUP_SIZE = 8;
+
+ // Capped at 15 (vectorSize=32768) because num_exceptions is uint16,
+ // so vectorSize must not exceed 65535 to avoid overflow when all values are exceptions.
+ static final int MAX_LOG_VECTOR_SIZE = 15;
+ static final int MIN_LOG_VECTOR_SIZE = 3;
+
+ static final int FLOAT_MAX_EXPONENT = 10;
+ static final int DOUBLE_MAX_EXPONENT = 18;
+
+ // Sampler constants matching C++ AlpConstants.
+ // Sample SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP vectors evenly distributed across a rowgroup
+ // of SAMPLER_ROWGROUP_SIZE values, then lock in top MAX_PRESET_COMBINATIONS combos.
+ static final int SAMPLER_ROWGROUP_SIZE = 122_880;
Review Comment:
Agreed, thank you. I went through each constant and sorted them by who actually reads them, in 78c1d182a.
The sampler constants (SAMPLER_ROWGROUP_SIZE, SAMPLER_SAMPLE_VECTORS_PER_ROWGROUP, MAX_PRESET_COMBINATIONS) are only used by the writer, so they moved to AlpValuesWriter. The rounding magic numbers and the powers of ten tables are only used by the codec, so they moved to AlpCodec.
What is left in AlpConstants is genuinely shared and all of it describes the wire format: the header and metadata sizes, the mode and encoding markers, the vector size bounds, and the per type exponent limits. It reads much better as a wire format definition than it did as a general bucket.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
Review Comment:
Happy to take this. Renamed to AlpCodec in 78c1d182a, along with its test. I went with AlpCodec over AlpUtil since it really is a codec rather than a collection of helpers.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpEncoderDecoder.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
Review Comment:
Understood, and sorry for the noise. All five files now use explicit static imports, in 78c1d182a. I did this last on purpose, after the constants had been moved to their new homes, so that the import lists reflect where things ended up rather than needing a second pass.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesWriter.java:
##########
@@ -0,0 +1,647 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.parquet.bytes.ByteBufferAllocator;
+import org.apache.parquet.bytes.BytesInput;
+import org.apache.parquet.bytes.BytesUtils;
+import org.apache.parquet.bytes.CapacityByteArrayOutputStream;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.values.ValuesWriter;
+import org.apache.parquet.column.values.bitpacking.BytePacker;
+import org.apache.parquet.column.values.bitpacking.BytePackerForLong;
+import org.apache.parquet.column.values.bitpacking.Packer;
+
+/**
+ * ALP (Adaptive Lossless floating-Point) values writer.
+ *
+ * <p>ALP encoding converts floating-point values to integers using decimal scaling,
+ * then applies Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Writing is incremental: values are buffered in a fixed-size vector buffer,
+ * and each full vector is encoded and flushed to the output stream immediately.
+ * On {@link #getBytes()}, any remaining partial vector is flushed, and the
+ * final page bytes are assembled.
+ *
+ * <p>Interleaved Page Layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
Review Comment:
Thanks for spotting that. The cause is that the cell contains `4B × numVectors`, and the HTML entity is seven characters in the source but renders as a single glyph, so the box could never line up in both your editor and the rendered javadoc at the same time.
Fixed in 78c1d182a by using the literal character and padding the cell to the right width, so every row is 73 characters in both views. The same diagram had been copied into AlpValuesReader with the same problem, so I fixed it there too.
##########
parquet-format-structures/pom.xml:
##########
@@ -66,6 +66,35 @@
</execution>
</executions>
</plugin>
+ <!--
Review Comment:
Your timing on this was good. #3709 merged a couple of days after you left the comment, so parquet.thrift is now inlined in the repo and I was able to take the approach you suggested.
In 229e0d99b, ALP = 10 is declared directly in the Encoding enum, and the perl script and the exec-maven-plugin execution that ran it are both gone. I verified the generated Encoding.java still carries ALP(10) after BYTE_STREAM_SPLIT(9) with no patch step involved.
There is one wrinkle I want your opinion on. dev/update-parquet-thrift.sh overwrites parquet.thrift wholesale from upstream, so the next time anyone runs it the ALP entry will disappear silently. It also rewrites the parquet-format.version sidecar, so a note there would not survive either. For now I have put a clearly marked notice in the enum itself saying it is a local addition that needs re-applying until ALP is accepted into parquet-format, on the grounds that at least the loss shows up in a diff. If you would prefer this handled a different way, such as a guard in the update script, I am glad to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r3942455381
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpCodec.java:
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.*;
+
+import org.apache.parquet.bytes.BytesUtils;
+
+/**
+ * Core ALP (Adaptive Lossless floating-Point) encoding and decoding logic.
+ *
+ * <p>ALP works by converting floating-point values to integers using decimal scaling,
+ * then applying Frame of Reference encoding and bit-packing.
+ * Values that cannot be losslessly converted are stored as exceptions.
+ *
+ * <p>Encoding formula: encoded = fastRound(value * POW10[e] * POW10_NEGATIVE[f])
+ * <p>Decoding formula: value = encoded * POW10[f] * POW10_NEGATIVE[e]
+ *
+ * <p>The order of operations is critical for IEEE 754 correctness. Both formulas must
+ * be evaluated as single expressions — storing the intermediate multiplication result
+ * in a variable before the second multiply changes IEEE 754 rounding and produces extra
+ * exceptions. Likewise, scaling uses multiply-by-reciprocal (via POW10_NEGATIVE) rather than
+ * division: this reproduces the exact IEEE 754 rounding of the ALP reference algorithm, so the
+ * encoded integers — and therefore which values become exceptions and the resulting bytes — are
+ * identical across implementations. It is about cross-implementation determinism, not any one
+ * language.
+ *
+ * <p>Exception conditions:
+ * <ul>
+ * <li>NaN values</li>
+ * <li>Infinity values</li>
+ * <li>Negative zero (-0.0)</li>
+ * <li>Out of integer range</li>
+ * <li>Round-trip failure (decode(encode(v)) != v)</li>
+ * </ul>
+ */
+final class AlpEncoderDecoder {
+
+ private static final double ENCODING_UPPER_LIMIT = 9223372036854774784.0;
+ private static final double ENCODING_LOWER_LIMIT = -9223372036854774784.0;
+ private static final float FLOAT_ENCODING_UPPER_LIMIT = 2147483520.0f;
+ private static final float FLOAT_ENCODING_LOWER_LIMIT = -2147483520.0f;
+
+ private AlpEncoderDecoder() {
+ // Utility class
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicFloatException(float value) {
+ if (Float.isNaN(value)) {
+ return true;
+ }
+ if (Float.isInfinite(value)) {
+ return true;
+ }
+ return Float.floatToRawIntBits(value) == FLOAT_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure. */
+ static boolean isFloatException(float value, int exponent, int factor) {
+ if (isIntrinsicFloatException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ float scaled = value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor];
+ if (!Float.isFinite(scaled) || scaled > FLOAT_ENCODING_UPPER_LIMIT || scaled < FLOAT_ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ int encoded = encodeFloat(value, exponent, factor);
+ float decoded = decodeFloat(encoded, exponent, factor);
+ return Float.floatToRawIntBits(value) != Float.floatToRawIntBits(decoded);
+ }
+
+ /** Round float to nearest integer using magic-number trick with sign branching. */
+ static int fastRoundFloat(float value) {
+ if (value >= 0) {
+ return (int) ((value + MAGIC_FLOAT) - MAGIC_FLOAT);
+ } else {
+ return (int) ((value - MAGIC_FLOAT) + MAGIC_FLOAT);
+ }
+ }
+
+ static int encodeFloat(float value, int exponent, int factor) {
+ return fastRoundFloat(value * FLOAT_POW10[exponent] * FLOAT_POW10_NEGATIVE[factor]);
+ }
+
+ static float decodeFloat(int encoded, int exponent, int factor) {
+ return encoded * FLOAT_POW10[factor] * FLOAT_POW10_NEGATIVE[exponent];
+ }
+
+ /** NaN, Inf, and -0.0 can never be encoded regardless of exponent/factor. */
+ static boolean isIntrinsicDoubleException(double value) {
+ if (Double.isNaN(value)) {
+ return true;
+ }
+ if (Double.isInfinite(value)) {
+ return true;
+ }
+ return Double.doubleToRawLongBits(value) == DOUBLE_NEGATIVE_ZERO_BITS;
+ }
+
+ /** Full exception check for a given (exponent, factor): intrinsic cases plus round-trip failure. */
+ static boolean isDoubleException(double value, int exponent, int factor) {
+ if (isIntrinsicDoubleException(value)) {
+ return true;
+ }
+ // Check before rounding: overflow or non-finite after scaling
+ double scaled = value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor];
+ if (!Double.isFinite(scaled) || scaled > ENCODING_UPPER_LIMIT || scaled < ENCODING_LOWER_LIMIT) {
+ return true;
+ }
+ long encoded = encodeDouble(value, exponent, factor);
+ double decoded = decodeDouble(encoded, exponent, factor);
+ return Double.doubleToRawLongBits(value) != Double.doubleToRawLongBits(decoded);
+ }
+
+ /** Round double to nearest integer using magic-number trick with sign branching. */
+ static long fastRoundDouble(double value) {
+ if (value >= 0) {
+ return (long) ((value + MAGIC_DOUBLE) - MAGIC_DOUBLE);
+ } else {
+ return (long) ((value - MAGIC_DOUBLE) + MAGIC_DOUBLE);
+ }
+ }
+
+ static long encodeDouble(double value, int exponent, int factor) {
+ return fastRoundDouble(value * DOUBLE_POW10[exponent] * DOUBLE_POW10_NEGATIVE[factor]);
+ }
+
+ static double decodeDouble(long encoded, int exponent, int factor) {
+ return encoded * DOUBLE_POW10[factor] * DOUBLE_POW10_NEGATIVE[exponent];
+ }
+
+ public static class EncodingParams {
+ final int exponent;
+ final int factor;
+ final int numExceptions;
+
+ EncodingParams(int exponent, int factor, int numExceptions) {
+ this.exponent = exponent;
+ this.factor = factor;
+ this.numExceptions = numExceptions;
+ }
+ }
+
+ // Index positions within an (exponent, factor) pair array.
+ private static final int E = 0;
+ private static final int F = 1;
+
+ // All valid (exponent, factor) pairs for the full search, precomputed in nested-loop order
+ // (e ascending, then f ascending) so the tie-break and early-exit behave exactly like an inline
+ // double loop. Reused across every vector to avoid per-call allocation.
+ private static final int[][] ALL_VALID_FLOAT_PAIRS = buildAllPairs(FLOAT_MAX_EXPONENT);
+ private static final int[][] ALL_VALID_DOUBLE_PAIRS = buildAllPairs(DOUBLE_MAX_EXPONENT);
+
+ private static int[][] buildAllPairs(int maxExponent) {
+ int count = (maxExponent + 1) * (maxExponent + 2) / 2;
+ int[][] pairs = new int[count][];
+ int idx = 0;
+ for (int e = 0; e <= maxExponent; e++) {
+ for (int f = 0; f <= e; f++) {
+ pairs[idx++] = new int[] {e, f};
+ }
+ }
+ return pairs;
+ }
+
+ /** Try all (exponent, factor) combos and pick the one with the smallest estimated compressed size. */
+ static EncodingParams findBestFloatParams(float[] values, int offset, int length) {
+ return pickBestFloat(values, offset, length, ALL_VALID_FLOAT_PAIRS);
+ }
+
+ /** Same as findBestFloatParams but only tries the cached preset combos. */
+ static EncodingParams findBestFloatParamsWithPresets(float[] values, int offset, int length, int[][] presets) {
+ return pickBestFloat(values, offset, length, presets);
+ }
+
+ /**
+ * Scores each (exponent, factor) pair by estimated compressed size and returns the best.
+ *
+ * <p>Estimated size (in bits) = {@code length * bitWidth + exceptions * (Float.SIZE + Short.SIZE)},
+ * where bitWidth is the number of bits needed to represent the signed range (max - min) of
+ * non-exception encoded values after frame-of-reference subtraction, matching the writer's FOR
+ * packing. This produces better compression ratios than minimizing exception count alone. Ties in
+ * size are broken toward the higher exponent, then the higher factor. The first pair (in {@code
+ * pairs} order) that encodes every value into a single FOR value with zero exceptions wins
+ * outright. When no pair yields any non-exception values, {@code pairs[0]} is returned.
+ */
+ private static EncodingParams pickBestFloat(float[] values, int offset, int length, int[][] pairs) {
+ int bestExponent = pairs[0][E];
+ int bestFactor = pairs[0][F];
+ int bestExceptions = length;
+ long bestEstimatedSize = Long.MAX_VALUE;
+
+ for (int[] pair : pairs) {
+ int e = pair[E];
+ int f = pair[F];
+ int exceptions = 0;
+ int minEncoded = Integer.MAX_VALUE;
+ int maxEncoded = Integer.MIN_VALUE;
+ for (int i = 0; i < length; i++) {
+ float value = values[offset + i];
+ if (isFloatException(value, e, f)) {
+ exceptions++;
+ } else {
+ int encoded = encodeFloat(value, e, f);
Review Comment:
You were right to ask, and it was worth doing. isFloatException has to encode the value in order to test whether it round trips, so calling it and then encodeFloat really did the work twice. The writer's per-vector loop was worse than the search loop you commented on: it ran the exception check, encoded again, and had a separate placeholder scan that encoded a third time.
Fixed in 8e743c60a. There are now tryEncodeFloat and tryEncodeDouble which give back the exception flag and the encoded value together. I went with a small caller-owned EncodeResult holder rather than an Optional, since an Optional would box on every value in what is the hottest loop in the writer. The scaled intermediate gets reused as well, so the two multiplies are also down to one.
Encoded bytes are unchanged. tryEncode uses the same expression in the same order as encodeFloat, which matters for cross implementation bit compatibility, and testTryEncodeMatchesSeparateExceptionCheckAndEncode asserts the new path agrees with the old pair on both the flag and the exact encoded bits across every valid (exponent, factor) pair.
On whether it was a performance concern, I want to give you the honest answer rather than a flattering one. Measured with AlpEncodingBenchmarks.writeDoubleAndFloatALP over a million rows, before was 197.9 +- 6.6 ms/op and after was 194.2 +- 2.9 ms/op. The error bars overlap, so at the level of writing a whole file this is within noise, because encoding is small next to compression and page assembly. The duplicated work was real and is gone, but I cannot claim a measurable end to end win from it.
##########
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInterOpReadAlp.java:
##########
@@ -0,0 +1,1408 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.parquet.hadoop;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.parquet.column.Encoding;
+import org.apache.parquet.column.ParquetProperties.WriterVersion;
+import org.apache.parquet.column.page.PageReadStore;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.example.data.simple.SimpleGroup;
+import org.apache.parquet.example.data.simple.convert.GroupRecordConverter;
+import org.apache.parquet.hadoop.example.ExampleParquetWriter;
+import org.apache.parquet.hadoop.metadata.CompressionCodecName;
+import org.apache.parquet.hadoop.metadata.ParquetMetadata;
+import org.apache.parquet.io.ColumnIOFactory;
+import org.apache.parquet.io.LocalInputFile;
+import org.apache.parquet.io.LocalOutputFile;
+import org.apache.parquet.io.MessageColumnIO;
+import org.apache.parquet.io.RecordReader;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Cross-compatibility test for ALP (Adaptive Lossless floating-Point) encoding.
+ *
+ * <p>Reads ALP-encoded parquet files generated by Arrow C++ and verifies that the Java
+ * implementation decodes them correctly.
+ *
+ * <p>Set ALP_TEST_FILE (env var or system property) to a single file, or use the
+ * ALP_TEST_DATA_DIR property pointing to the alp-test-data/ directory.
+ *
+ * @see <a href="https://github.com/apache/arrow/pull/48345">Arrow C++ ALP PR</a>
+ * @see <a href="https://github.com/apache/parquet-testing/pull/100">parquet-testing ALP PR</a>
+ */
+public class TestInterOpReadAlp {
+ private static final Logger LOG = LoggerFactory.getLogger(TestInterOpReadAlp.class);
+
+ @TempDir
+ java.nio.file.Path temp;
+
+ /** Mirrors JUnit 4's {@code TemporaryFolder#newFolder()}: a fresh directory per call. */
+ private File newFolder() throws IOException {
+ return Files.createTempDirectory(temp, "junit").toFile();
+ }
+
+ private static final String[] CPP_DOUBLE_FILES = {"alp_spotify1.parquet", "alp_arade.parquet"};
+ private static final String[] CPP_FLOAT_FILES = {"alp_float_spotify1.parquet", "alp_float_arade.parquet"};
+
+ private java.nio.file.Path getTestDataDir() {
+ String dir = System.getProperty("ALP_TEST_DATA_DIR");
Review Comment:
You're right, and I would like to move this over to the standard harness. The blocker is that the ALP fixture files are not in parquet-testing yet. They are proposed in https://github.com/apache/parquet-testing/pull/100, which is still open.
Once that merges I will convert this to use InterOpTester.GetInterOpFile with a pinned changeset, the same as TestInterOpReadByteStreamSplit and the other interop tests. Until then the test reads from a local directory and skips when it is not present, which does mean it is effectively a no-op in CI, so I understand it is not carrying its weight right now.
If you have any influence over getting parquet-testing #100 looked at, that would help a lot. Cross language read back is the part of this work I am least able to demonstrate on my own, so it matters for the vote more than for this test alone.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/factory/DefaultV1ValuesWriterFactory.java:
##########
@@ -147,7 +148,13 @@ private ValuesWriter getInt96ValuesWriter(ColumnDescriptor path) {
private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) {
final ValuesWriter fallbackWriter;
- if (this.parquetProperties.isByteStreamSplitEnabled(path)) {
+ if (this.parquetProperties.isAlpEnabled(path)) {
Review Comment:
Thanks for weighing in on this, it is helpful to have a second opinion since I went back and forth on it. I will leave ALP registered in both V1 and V2 then, and I agree that staying consistent with how isByteStreamSplitEnabled is handled is the least surprising option. If the writer version work lands later and there is a reason to pin ALP to a particular version, that seems like a better time to revisit it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "vinooganesh (via GitHub)" <gi...@apache.org>.
vinooganesh commented on PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#issuecomment-5650398149
@RussellSpitzer thanks for the review and the approval. Three of the four nits are done, and I am still working through the thrift one.
**Java docs** — f7f7273aa. The one you spotted described the build-time patch that was deleted in 229e0d99b, so I dropped the detail rather than updating it. I grepped for anything else mentioning the patch or a local addition and that was the only one left.
**Skips** — the `testEnumEquivalence` skip is gone in 0fcd73e98, so ALP round trips like every other encoding now. One judgement call for you: `TestInterOpReadAlp` also skips when pyarrow reports "Unknown encoding type". That is a skip about ALP not existing, but in pyarrow rather than here, and removing it makes the test fail on most machines. Happy to pull it if you would rather it fail loudly.
**`withoutAlp(column)`** — a8c584f64. It was missing because ColumnProperty treated a per-column null as unset and fell back to the default, so it checks containsKey now and an explicit null overrides; every other per-column property either cannot hold a null or already defaults to one, so nothing else changes. Two fixes had to come with it: the schema check was reading "named in the ALP property" as "enabled", so disabling ALP on an int column failed with a message claiming ALP was enabled on it, and toString rendered a disabled column as null, which undid your other nit.
**Thrift** — still looking into it. There is a wrinkle in doing it as a precursor PR that I want to understand properly before I propose something, so I will follow up on that thread.
Your approval is against e660c6cec, so it misses the four commits since. Another look whenever you get a chance would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] Parquet Java ALP Implementation [parquet-java]
Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #3397:
URL: https://github.com/apache/parquet-java/pull/3397#discussion_r4099677508
##########
parquet-column/src/main/java/org/apache/parquet/column/values/alp/AlpValuesReader.java:
##########
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.column.values.alp;
+
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_COMPRESSION_MODE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_HEADER_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INFO_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.ALP_INTEGER_ENCODING_FOR;
+import static org.apache.parquet.column.values.alp.AlpConstants.MAX_LOG_VECTOR_SIZE;
+import static org.apache.parquet.column.values.alp.AlpConstants.MIN_LOG_VECTOR_SIZE;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
+import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.column.values.ValuesReader;
+import org.apache.parquet.io.ParquetDecodingException;
+
+/**
+ * Abstract base class for ALP values readers with lazy per-vector decoding.
+ *
+ * <p>Reads ALP-encoded values from the interleaved page layout:
+ * <pre>
+ * ┌─────────┬──────────────────────┬──────────────┬──────────────┬─────┐
+ * │ Header │ Offset Array │ Vector 0 │ Vector 1 │ ... │
+ * │ 7 bytes │ 4B × numVectors │ (interleaved)│ (interleaved)│ │
+ * └─────────┴──────────────────────┴──────────────┴──────────────┴─────┘
+ * </pre>
+ *
+ * <p>Each vector is decoded lazily on first access. Skipping values does not
+ * trigger decoding of intermediate vectors.
+ */
+abstract class AlpValuesReader extends ValuesReader {
+
+ protected int vectorSize;
+ protected int totalCount;
+ protected int numVectors;
+ protected int pageValueIndex;
+ protected int currentVectorNumber;
+
+ protected int[] vectorOffsets;
+ protected ByteBuffer vectorsData;
+ protected int offsetArraySize;
+
+ // Scratch buffer for exception positions within a vector; shared by both readers (int[] in each).
+ protected int[] excPositionsBuffer;
+
+ AlpValuesReader() {
+ this.pageValueIndex = 0;
+ this.totalCount = 0;
+ this.currentVectorNumber = -1;
+ }
+
+ @Override
+ public void initFromPage(int valuesCount, ByteBufferInputStream stream)
+ throws ParquetDecodingException, IOException {
Review Comment:
I think this should be broken down into helper methods. It's pretty large and complex.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org