You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "iemejia (via GitHub)" <gi...@apache.org> on 2026/05/17 22:38:35 UTC
[PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
iemejia opened a new pull request, #3565:
URL: https://github.com/apache/parquet-java/pull/3565
Part of #3530 — Apache Parquet Java Performance Improvements
## Summary
Replace `ByteBufferInputStream` and `LittleEndianDataInputStream` wrappers with direct `ByteBuffer` access for all PLAIN value readers and writers.
**Readers** (`PlainValuesReader`, `BooleanPlainValuesReader`, `BinaryPlainValuesReader`, `FixedLenByteArrayPlainValuesReader`): hold a little-endian `ByteBuffer` from `initFromPage()` and call `getInt`/`getLong`/`getFloat`/`getDouble` directly, eliminating per-value stream overhead.
**Writers** (`PlainValuesWriter`, `BooleanPlainValuesWriter`, `FixedLenByteArrayPlainValuesWriter`): write through `CapacityByteArrayOutputStream`'s new `writeInt`/`writeLong` methods which put values directly into the NIO slab buffer in little-endian order, avoiding temporary byte-array allocation.
**Supporting changes**:
- `CapacityByteArrayOutputStream`: allocate slabs with `ByteOrder.LITTLE_ENDIAN`, add `writeInt(int)` and `writeLong(long)` for single-value NIO writes.
- `BytesInput`: add zero-copy `writeTo(ByteBuffer)` and `toByteArray()` using bulk `ByteBuffer.get()` instead of stream copy.
- `LittleEndianDataOutputStream`: batch single-byte writes into single `write(buf, 0, N)` calls for `writeShort`/`writeInt`.
Includes JMH benchmarks (`PlainEncodingBenchmark`, `PlainDecodingBenchmark`) covering all 7 primitive types for both encoding and decoding.
## Benchmark results
**Environment**: JDK 25.0.3 (Temurin), OpenJDK 64-Bit Server VM, JMH 1.37, Linux x86_64.
Decoding (100K values/iteration, 3 forks x 5 iterations, throughput mode):
| Benchmark | Master (M ops/s) | Branch (M ops/s) | Speedup |
|---|---:|---:|---:|
| decodeInt | 425 | 5,427 | **12.8x** |
| decodeFloat | 416 | 5,440 | **13.1x** |
| decodeLong | 119 | 4,720 | **39.5x** (\*) |
| decodeDouble | 116 | 6,026 | **51.8x** (\*) |
| decodeBoolean | 639 | 1,642 | **2.6x** |
| decodeFlba (len=2,12,16) | 188 | 680 | **3.6x** |
| decodeBinary (len=10,100,1000) | 142 | 225-230 | **1.6x** |
Encoding:
| Benchmark | Master (M ops/s) | Branch (M ops/s) | Speedup |
|---|---:|---:|---:|
| encodeInt | 148 | 559 | **3.8x** |
| encodeFloat | 150 | 532 | **3.5x** |
| encodeLong | 193 | 478 | **2.5x** |
| encodeDouble | 179 | 439 | **2.4x** |
| encodeBoolean | 850 | 1,692 | **2.0x** |
| encodeBinary (len=10) | 76 | 150 | **2.0x** |
| encodeFlba (len=2-16) | 156-184 | 178-224 | **1.1-1.2x** |
(\*) decodeLong/Double show JIT variance across forks (error bars >20%); true steady-state likely ~13x consistent with INT32/FLOAT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3501910954
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,56 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private static final ByteBuffer EMPTY_LE_BUFFER =
+ ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer();
+
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
+ int length = buffer.getInt();
+ ByteBuffer valueSlice = buffer.slice();
+ valueSlice.limit(length);
+ buffer.position(buffer.position() + length);
+ return Binary.fromConstantByteBuffer(valueSlice);
+ } catch (RuntimeException e) {
+ throw new ParquetDecodingException("could not read bytes at offset " + buffer.position(), e);
}
}
@Override
public void skip() {
try {
- int length = BytesUtils.readIntLittleEndian(in);
- in.skipFully(length);
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not skip bytes at offset " + in.position(), e);
+ int length = buffer.getInt();
+ buffer.position(buffer.position() + length);
+ } catch (RuntimeException e) {
Review Comment:
The original code caught `IOException | RuntimeException` because it read through `ByteBufferInputStream` / `BytesUtils.readIntLittleEndian(in)` which throw checked `IOException`. The new code uses `ByteBuffer.getInt()` and `ByteBuffer.position()` directly -- these only throw `RuntimeException` subclasses (`BufferUnderflowException`, `IllegalArgumentException`), so `IOException` is no longer possible. Adding `catch (IOException e)` would actually be a compile error since nothing in the try block declares it. The same `catch (RuntimeException e)` pattern is used consistently across all the other readers (`PlainValuesReader`, `FixedLenByteArrayPlainValuesReader`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3343039837
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
Review Comment:
Good point. The old `protected LittleEndianDataInputStream in` field is now `protected ByteBuffer buffer` — the type change is binary-incompatible regardless, so a deprecation cycle wouldn't help external subclasses (they'd get a compile error either way). I searched the project and only internal subclasses (the 4 inner classes) access this field. I think this is acceptable given this class was never annotated `@Public` and the field type change makes deprecation impractical. WDYT?
##########
parquet-common/src/main/java/org/apache/parquet/bytes/BytesInput.java:
##########
@@ -643,6 +673,20 @@ public ByteBuffer toByteBuffer() throws IOException {
return java.nio.ByteBuffer.wrap(in, offset, length);
}
+ /**
+ * Zero-copy override: returns the backing array directly when fully used,
+ * skipping the base-class BAOS allocation + copy on every decompressor call.
+ * Returning the mutable array is safe — the base class already exposes a
+ * mutable {@code BAOS.getBuf()}.
+ */
+ @Override
+ public byte[] toByteArray() {
Review Comment:
Addressed in f0bdac633. The base-class `toInputStream()` now tries `getInternalByteBuffer()` first (zero-copy fast path) before falling back to the deprecated `toByteBuffer()`. Also added `getInternalByteBuffer()` and `toInputStream()` overrides to `ByteArrayBytesInput` so the byte-array-backed path is zero-copy too. Added `@SuppressWarnings("deprecation")` on the intentional overrides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3291176269
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,44 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
Review Comment:
Should we keep the `ParquetDecodingException`? Otherwise we're throwing the raw `{IOException,RuntimeException}` which is a behavioral change.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
+ protected ByteBuffer buffer;
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in = new LittleEndianDataInputStream(stream.remainingStream());
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
+ }
}
@Override
public void skip() {
skip(1);
}
- void skipBytesFully(int n) throws IOException {
- int skipped = 0;
- while (skipped < n) {
- skipped += in.skipBytes(n - skipped);
- }
- }
-
public static class DoublePlainValuesReader extends PlainValuesReader {
@Override
public void skip(int n) {
- try {
- skipBytesFully(n * 8);
- } catch (IOException e) {
- throw new ParquetDecodingException("could not skip " + n + " double values", e);
Review Comment:
Same here, do we want to keep the `ParquetDecodingException`?
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
+ protected ByteBuffer buffer;
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in = new LittleEndianDataInputStream(stream.remainingStream());
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
+ }
}
@Override
public void skip() {
skip(1);
}
- void skipBytesFully(int n) throws IOException {
- int skipped = 0;
- while (skipped < n) {
- skipped += in.skipBytes(n - skipped);
- }
- }
-
public static class DoublePlainValuesReader extends PlainValuesReader {
@Override
public void skip(int n) {
- try {
- skipBytesFully(n * 8);
- } catch (IOException e) {
- throw new ParquetDecodingException("could not skip " + n + " double values", e);
- }
+ buffer.position(buffer.position() + n * 8);
Review Comment:
Should we use `Math.multiplyExact` here and below?
##########
parquet-common/src/main/java/org/apache/parquet/bytes/BytesInput.java:
##########
@@ -643,6 +673,20 @@ public ByteBuffer toByteBuffer() throws IOException {
return java.nio.ByteBuffer.wrap(in, offset, length);
}
+ /**
+ * Zero-copy override: returns the backing array directly when fully used,
+ * skipping the base-class BAOS allocation + copy on every decompressor call.
+ * Returning the mutable array is safe — the base class already exposes a
+ * mutable {@code BAOS.getBuf()}.
+ */
+ @Override
+ public byte[] toByteArray() {
Review Comment:
This overrides a deprecated API, as a follow-up we probably should move the internal calls to the new API:
```java
@deprecated Use {@link #toByteBuffer(ByteBufferAllocator, Consumer)}
```
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,44 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
- }
+ int length = buffer.getInt();
+ ByteBuffer valueSlice = buffer.slice();
+ valueSlice.limit(length);
+ buffer.position(buffer.position() + length);
+ return Binary.fromConstantByteBuffer(valueSlice);
}
@Override
public void skip() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- in.skipFully(length);
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not skip bytes at offset " + in.position(), e);
- }
+ int length = buffer.getInt();
+ buffer.position(buffer.position() + length);
}
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
- LOG.debug(
- "init from page at offset {} for length {}",
- stream.position(),
- (stream.available() - stream.position()));
- this.in = stream.remainingStream();
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
Review Comment:
Should we create a constant for the `ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);`?
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
Review Comment:
We should go through the deprecation cycle here, but is anything using this outside of the project itself?
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/FixedLenByteArrayPlainValuesReader.java:
##########
@@ -19,52 +19,56 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
import org.apache.parquet.bytes.ByteBufferInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* ValuesReader for FIXED_LEN_BYTE_ARRAY.
+ *
+ * <p>Reads directly from a {@link ByteBuffer}, bypassing the {@link ByteBufferInputStream}
+ * wrapper to avoid per-value stream overhead (remaining checks, IOException wrapping,
+ * virtual dispatch). The underlying page data is obtained as a single contiguous
+ * {@link ByteBuffer} via {@link ByteBufferInputStream#slice(int)}.
*/
public class FixedLenByteArrayPlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(FixedLenByteArrayPlainValuesReader.class);
private final int length;
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
public FixedLenByteArrayPlainValuesReader(int length) {
this.length = length;
}
@Override
public Binary readBytes() {
- try {
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
Review Comment:
Same as above, should we keep the wrapped `ParquetDecodingException`?
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BooleanPlainValuesReader.java:
##########
@@ -18,57 +18,65 @@
*/
package org.apache.parquet.column.values.plain;
-import static org.apache.parquet.column.values.bitpacking.Packer.LITTLE_ENDIAN;
-
import java.io.IOException;
+import java.nio.ByteBuffer;
import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.column.values.bitpacking.ByteBitPackingValuesReader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * encodes boolean for the plain encoding: one bit at a time (0 = false)
+ * Decodes PLAIN-encoded booleans: one bit per value, packed 8 per byte, little-endian
+ * bit order (bit 0 of each byte is the first value).
+ *
+ * <p>Direct bit extraction from the page ByteBuffer avoids the overhead of the generic
+ * bit-packing machinery ({@code ByteBitPackingValuesReader}) and intermediate
+ * {@code int[8]} buffers.
*/
public class BooleanPlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(BooleanPlainValuesReader.class);
- private ByteBitPackingValuesReader in = new ByteBitPackingValuesReader(1, LITTLE_ENDIAN);
+ private byte[] pageData;
+ private int pageOffset;
+ private int bitIndex;
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#readBoolean()
- */
@Override
- public boolean readBoolean() {
- return in.readInteger() == 0 ? false : true;
+ public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
+ LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
+ int effectiveBitLength = valueCount; // bitWidth = 1
+ int length = BytesUtils.paddedByteCountFromBits(effectiveBitLength);
+ length = Math.min(length, stream.available());
+ ByteBuffer buf = stream.slice(length);
+
+ // Bulk access: use backing array directly if available, otherwise copy once.
+ if (buf.hasArray()) {
+ pageData = buf.array();
+ pageOffset = buf.arrayOffset() + buf.position();
+ } else {
+ pageData = new byte[length];
+ buf.get(pageData);
+ pageOffset = 0;
+ }
+ bitIndex = 0;
+ updateNextOffset(length);
}
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#skip()
- */
@Override
- public void skip() {
- in.readInteger();
+ public boolean readBoolean() {
+ int byteIdx = pageOffset + (bitIndex >>> 3);
+ int bitPos = bitIndex & 7;
+ bitIndex++;
+ return ((pageData[byteIdx] >>> bitPos) & 1) != 0;
}
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#initFromPage(int, ByteBufferInputStream)
- */
@Override
- public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
- LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in.initFromPage(valueCount, stream);
+ public void skip() {
+ bitIndex++;
}
- @Deprecated
@Override
- public int getNextOffset() {
- return in.getNextOffset();
+ public void skip(int n) {
+ bitIndex += n;
Review Comment:
Should we check for bounds, and throw a `ParquetDecodingException` in case of out of bounds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko merged PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3501876414
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,56 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private static final ByteBuffer EMPTY_LE_BUFFER =
+ ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer();
+
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
+ int length = buffer.getInt();
+ ByteBuffer valueSlice = buffer.slice();
+ valueSlice.limit(length);
+ buffer.position(buffer.position() + length);
+ return Binary.fromConstantByteBuffer(valueSlice);
+ } catch (RuntimeException e) {
+ throw new ParquetDecodingException("could not read bytes at offset " + buffer.position(), e);
}
}
@Override
public void skip() {
try {
- int length = BytesUtils.readIntLittleEndian(in);
- in.skipFully(length);
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not skip bytes at offset " + in.position(), e);
+ int length = buffer.getInt();
+ buffer.position(buffer.position() + length);
+ } catch (RuntimeException e) {
Review Comment:
Should we include IOException as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3343038071
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/FixedLenByteArrayPlainValuesReader.java:
##########
@@ -19,52 +19,56 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
import org.apache.parquet.bytes.ByteBufferInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* ValuesReader for FIXED_LEN_BYTE_ARRAY.
+ *
+ * <p>Reads directly from a {@link ByteBuffer}, bypassing the {@link ByteBufferInputStream}
+ * wrapper to avoid per-value stream overhead (remaining checks, IOException wrapping,
+ * virtual dispatch). The underlying page data is obtained as a single contiguous
+ * {@link ByteBuffer} via {@link ByteBufferInputStream#slice(int)}.
*/
public class FixedLenByteArrayPlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(FixedLenByteArrayPlainValuesReader.class);
private final int length;
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
public FixedLenByteArrayPlainValuesReader(int length) {
this.length = length;
}
@Override
public Binary readBytes() {
- try {
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
Review Comment:
Done. Added try/catch wrapping in `readBytes()`, `skip()`, and `skip(int n)`. Also using `Math.multiplyExact(n, length)` in `skip(int n)` to detect overflow.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
+ protected ByteBuffer buffer;
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in = new LittleEndianDataInputStream(stream.remainingStream());
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
+ }
}
@Override
public void skip() {
skip(1);
}
- void skipBytesFully(int n) throws IOException {
- int skipped = 0;
- while (skipped < n) {
- skipped += in.skipBytes(n - skipped);
- }
- }
-
public static class DoublePlainValuesReader extends PlainValuesReader {
@Override
public void skip(int n) {
- try {
- skipBytesFully(n * 8);
- } catch (IOException e) {
- throw new ParquetDecodingException("could not skip " + n + " double values", e);
Review Comment:
Done. All `readXxx()` and `skip(int n)` methods across DoublePlainValuesReader, FloatPlainValuesReader, IntegerPlainValuesReader, and LongPlainValuesReader now wrap `RuntimeException` in `ParquetDecodingException` with descriptive messages matching the original error contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#issuecomment-4725304164
@Fokko another one ready for a last review / approval :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3343041130
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BooleanPlainValuesReader.java:
##########
@@ -18,57 +18,65 @@
*/
package org.apache.parquet.column.values.plain;
-import static org.apache.parquet.column.values.bitpacking.Packer.LITTLE_ENDIAN;
-
import java.io.IOException;
+import java.nio.ByteBuffer;
import org.apache.parquet.bytes.ByteBufferInputStream;
+import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.column.values.bitpacking.ByteBitPackingValuesReader;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * encodes boolean for the plain encoding: one bit at a time (0 = false)
+ * Decodes PLAIN-encoded booleans: one bit per value, packed 8 per byte, little-endian
+ * bit order (bit 0 of each byte is the first value).
+ *
+ * <p>Direct bit extraction from the page ByteBuffer avoids the overhead of the generic
+ * bit-packing machinery ({@code ByteBitPackingValuesReader}) and intermediate
+ * {@code int[8]} buffers.
*/
public class BooleanPlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(BooleanPlainValuesReader.class);
- private ByteBitPackingValuesReader in = new ByteBitPackingValuesReader(1, LITTLE_ENDIAN);
+ private byte[] pageData;
+ private int pageOffset;
+ private int bitIndex;
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#readBoolean()
- */
@Override
- public boolean readBoolean() {
- return in.readInteger() == 0 ? false : true;
+ public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
+ LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
+ int effectiveBitLength = valueCount; // bitWidth = 1
+ int length = BytesUtils.paddedByteCountFromBits(effectiveBitLength);
+ length = Math.min(length, stream.available());
+ ByteBuffer buf = stream.slice(length);
+
+ // Bulk access: use backing array directly if available, otherwise copy once.
+ if (buf.hasArray()) {
+ pageData = buf.array();
+ pageOffset = buf.arrayOffset() + buf.position();
+ } else {
+ pageData = new byte[length];
+ buf.get(pageData);
+ pageOffset = 0;
+ }
+ bitIndex = 0;
+ updateNextOffset(length);
}
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#skip()
- */
@Override
- public void skip() {
- in.readInteger();
+ public boolean readBoolean() {
+ int byteIdx = pageOffset + (bitIndex >>> 3);
+ int bitPos = bitIndex & 7;
+ bitIndex++;
+ return ((pageData[byteIdx] >>> bitPos) & 1) != 0;
}
- /**
- * {@inheritDoc}
- *
- * @see org.apache.parquet.column.values.ValuesReader#initFromPage(int, ByteBufferInputStream)
- */
@Override
- public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
- LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in.initFromPage(valueCount, stream);
+ public void skip() {
+ bitIndex++;
}
- @Deprecated
@Override
- public int getNextOffset() {
- return in.getNextOffset();
+ public void skip(int n) {
+ bitIndex += n;
Review Comment:
Done. Added a `bitCount` field (set to `length * 8` in `initFromPage`) and an explicit bounds check in `readBoolean()` that throws `ParquetDecodingException` with a descriptive message when attempting to read beyond the page boundary.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/PlainValuesReader.java:
##########
@@ -19,120 +19,91 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.LittleEndianDataInputStream;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
- * Plain encoding for float, double, int, long
+ * Plain encoding for float, double, int, long.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * bypassing the deprecated {@code LittleEndianDataInputStream} wrapper to avoid per-value virtual
+ * dispatch overhead. The underlying page data is obtained as a single contiguous {@link ByteBuffer}
+ * via {@link ByteBufferInputStream#slice(int)}.
*/
public abstract class PlainValuesReader extends ValuesReader {
private static final Logger LOG = LoggerFactory.getLogger(PlainValuesReader.class);
- protected LittleEndianDataInputStream in;
+ protected ByteBuffer buffer;
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
LOG.debug("init from page at offset {} for length {}", stream.position(), stream.available());
- this.in = new LittleEndianDataInputStream(stream.remainingStream());
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
+ }
}
@Override
public void skip() {
skip(1);
}
- void skipBytesFully(int n) throws IOException {
- int skipped = 0;
- while (skipped < n) {
- skipped += in.skipBytes(n - skipped);
- }
- }
-
public static class DoublePlainValuesReader extends PlainValuesReader {
@Override
public void skip(int n) {
- try {
- skipBytesFully(n * 8);
- } catch (IOException e) {
- throw new ParquetDecodingException("could not skip " + n + " double values", e);
- }
+ buffer.position(buffer.position() + n * 8);
Review Comment:
Done. Applied `Math.multiplyExact` in all `skip(int n)` methods: `Math.multiplyExact(n, 8)` for double/long, `Math.multiplyExact(n, 4)` for float/int, and `Math.multiplyExact(n, length)` in `FixedLenByteArrayPlainValuesReader`. Overflow now produces an `ArithmeticException` which gets caught and wrapped in `ParquetDecodingException`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on code in PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#discussion_r3343036596
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,44 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
Review Comment:
Done. Added try/catch wrapping `RuntimeException` (which covers `BufferUnderflowException`, `IllegalArgumentException`, etc.) into `ParquetDecodingException` in both `readBytes()` and `skip()`.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/plain/BinaryPlainValuesReader.java:
##########
@@ -19,44 +19,44 @@
package org.apache.parquet.column.values.plain;
import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.ByteOrder;
import org.apache.parquet.bytes.ByteBufferInputStream;
-import org.apache.parquet.bytes.BytesUtils;
import org.apache.parquet.column.values.ValuesReader;
-import org.apache.parquet.io.ParquetDecodingException;
import org.apache.parquet.io.api.Binary;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+/**
+ * Plain encoding reader for BINARY values.
+ *
+ * <p>Reads directly from a {@link ByteBuffer} with {@link ByteOrder#LITTLE_ENDIAN} byte order,
+ * using {@link ByteBuffer#getInt()} for the 4-byte length prefix instead of 4 individual
+ * {@code InputStream.read()} calls through {@link org.apache.parquet.bytes.BytesUtils#readIntLittleEndian}.
+ */
public class BinaryPlainValuesReader extends ValuesReader {
- private static final Logger LOG = LoggerFactory.getLogger(BinaryPlainValuesReader.class);
- private ByteBufferInputStream in;
+ private ByteBuffer buffer;
@Override
public Binary readBytes() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- return Binary.fromConstantByteBuffer(in.slice(length));
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not read bytes at offset " + in.position(), e);
- }
+ int length = buffer.getInt();
+ ByteBuffer valueSlice = buffer.slice();
+ valueSlice.limit(length);
+ buffer.position(buffer.position() + length);
+ return Binary.fromConstantByteBuffer(valueSlice);
}
@Override
public void skip() {
- try {
- int length = BytesUtils.readIntLittleEndian(in);
- in.skipFully(length);
- } catch (IOException | RuntimeException e) {
- throw new ParquetDecodingException("could not skip bytes at offset " + in.position(), e);
- }
+ int length = buffer.getInt();
+ buffer.position(buffer.position() + length);
}
@Override
public void initFromPage(int valueCount, ByteBufferInputStream stream) throws IOException {
- LOG.debug(
- "init from page at offset {} for length {}",
- stream.position(),
- (stream.available() - stream.position()));
- this.in = stream.remainingStream();
+ int available = stream.available();
+ if (available > 0) {
+ this.buffer = stream.slice(available).order(ByteOrder.LITTLE_ENDIAN);
+ } else {
+ this.buffer = ByteBuffer.allocate(0).order(ByteOrder.LITTLE_ENDIAN);
Review Comment:
Done. Extracted `EMPTY_LE_BUFFER` as a `private static final` read-only constant (`ByteBuffer.allocate(0).order(LITTLE_ENDIAN).asReadOnlyBuffer()`), used via `.duplicate()` in `initFromPage`. Same pattern applied in `PlainValuesReader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#issuecomment-4948708220
Let's move this forward, thanks @iemejia for working on this! This is an impressive speed-up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [PR] GH-3530: Optimize PLAIN encoding and decoding with direct ByteBuffer I/O [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on PR #3565:
URL: https://github.com/apache/parquet-java/pull/3565#issuecomment-4948944154
Thank you @Fokko ! Great to see this landing and benefitting everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org