You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "iemejia (via GitHub)" <gi...@apache.org> on 2026/04/19 12:09:15 UTC
[I] Optimize PlainValuesWriter by writing directly to ByteBuffer slabs (up to 2x encode speedup) [parquet-java]
iemejia opened a new issue, #3495:
URL: https://github.com/apache/parquet-java/issues/3495
### Describe the enhancement requested
`PlainValuesWriter` (used for PLAIN-encoded INT32, INT64, FLOAT, DOUBLE, and BINARY
columns) currently writes each value through two layers of abstraction:
```
PlainValuesWriter -> LittleEndianDataOutputStream -> CapacityByteArrayOutputStream
```
Per `writeInt()`, `LittleEndianDataOutputStream` decomposes the int into 4 bytes
in a temporary `writeBuffer[8]` array and calls `out.write(writeBuffer, 0, 4)`,
which dispatches through the `OutputStream` chain into `CapacityByteArrayOutputStream`.
That path performs:
- 4 byte-shift operations for little-endian decomposition
- 1 intermediate `writeBuffer[8]` array write
- 2 levels of virtual dispatch
- 1 bounds check in `write(byte[], off, len)`
- 1 `System.arraycopy` for 4 bytes
Since `CapacityByteArrayOutputStream` already buffers into `ByteBuffer` slabs
internally, the entire chain can be collapsed into a single `ByteBuffer.putInt()`
call, which is a HotSpot intrinsic that compiles to a single unaligned store on
x86/ARM when the buffer is in `LITTLE_ENDIAN` order.
### Proposal
1. In `CapacityByteArrayOutputStream`:
- Set `ByteOrder.LITTLE_ENDIAN` on newly allocated slabs in `addSlab()`.
- Add `writeInt(int)` and `writeLong(long)` methods that call
`currentSlab.putInt(v)` / `currentSlab.putLong(v)` directly, with a single
remaining-check that grows the slab if needed.
2. In `PlainValuesWriter`:
- Remove the `LittleEndianDataOutputStream` field entirely.
- `writeInteger(v)` -> `arrayOut.writeInt(v)`
- `writeLong(v)` -> `arrayOut.writeLong(v)`
- `writeFloat(v)` -> `arrayOut.writeInt(Float.floatToIntBits(v))`
- `writeDouble(v)` -> `arrayOut.writeLong(Double.doubleToLongBits(v))`
- `writeBytes(Binary v)` -> `arrayOut.writeInt(v.length()); v.writeTo(arrayOut);`
- `getBytes()` no longer needs to flush a buffering layer.
- `close()` no longer closes the defunct stream.
What was eliminated per `writeInt` call:
- 4 byte-shift operations for little-endian decomposition
- 1 intermediate `writeBuffer[8]` array write
- 2 levels of virtual dispatch
- 1 bounds check in `write(byte[], off, len)`
- 1 `System.arraycopy` for 4 bytes
Replaced with:
- 1 remaining-check on the slab `ByteBuffer`
- 1 `ByteBuffer.putInt()` call (single JVM intrinsic, ~1 store instruction on
little-endian architectures)
### Benchmark results
`IntEncodingBenchmark.encodePlain` (100,000 INT32 values per invocation, JMH
`-wi 3 -i 5 -f 1`):
| Pattern | Before (ops/s) | After (ops/s) | Improvement |
|------------------|---------------:|--------------:|------------:|
| SEQUENTIAL | 26,817,451 | 52,953,193 | **+97.5% (2.0x)** |
| RANDOM | 28,517,312 | 37,774,036 | **+32.5%** |
| LOW_CARDINALITY | 28,705,158 | 52,819,678 | **+84.0%** |
| HIGH_CARDINALITY | 28,595,519 | 37,862,571 | **+32.4%** |
The improvement varies by pattern: SEQUENTIAL and LOW_CARDINALITY see ~2x because
the slab `putInt()` path has highly predictable branching (slab rarely runs out
for sequential writes). RANDOM and HIGH_CARDINALITY still see a solid +32%
improvement.
The same code path also benefits `writeLong()`, `writeFloat()`, `writeDouble()`,
and the length prefix written by `writeBytes(Binary)`.
Decode round-trip verified: re-reading the encoded data with `PlainValuesReader`
produces identical values at ~1.15B ops/s.
### Validation
All 573 `parquet-column` tests and 308 `parquet-common` tests pass with the
change applied.
### Component(s)
Core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org
Re: [I] Optimize PlainValuesWriter by writing directly to ByteBuffer slabs (up to 2x encode speedup) [parquet-java]
Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia closed issue #3495: Optimize PlainValuesWriter by writing directly to ByteBuffer slabs (up to 2x encode speedup)
URL: https://github.com/apache/parquet-java/issues/3495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org