You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "iemejia (via GitHub)" <gi...@apache.org> on 2026/04/21 16:47:00 UTC

[I] Optimize LittleEndianDataOutputStream.writeInt/writeShort with bulk write (resolves existing TODO) [parquet-java]

iemejia opened a new issue, #3518:
URL: https://github.com/apache/parquet-java/issues/3518

   ## Background
   
   `LittleEndianDataOutputStream.writeInt(int)` and `writeShort(int)` decompose the value byte-by-byte and call `out.write(int)` for each byte:
   
   ```java
   public final void writeInt(int v) throws IOException {
     out.write((v >>> 0) & 0xFF);
     out.write((v >>> 8) & 0xFF);
     out.write((v >>> 16) & 0xFF);
     out.write((v >>> 24) & 0xFF);
   }
   ```
   
   When the underlying stream is `CapacityByteArrayOutputStream` (the typical case in Parquet writers), each `out.write(int)` performs a `hasRemaining` check, a `Math.addExact` for the new size, possibly a slab-grow check, and a single-byte store. For `writeInt`, that's **4 separate trips** through the bookkeeping.
   
   The class already has the right pattern in `writeLong`: build a `writeBuffer[]` and emit a single `out.write(writeBuffer, 0, 8)`. The buffer is even pre-allocated for that purpose. `writeInt` and `writeShort` just don't use it.
   
   There's a `TODO` comment in `writeInt` (lines 147–149 in current master) acknowledging this:
   
   ```
   // TODO: see note in LittleEndianDataInputStream: maybe faster
   // to use Integer.reverseBytes() and then writeInt, or a ByteBuffer
   // approach
   ```
   
   ## Proposal
   
   Extend the existing `writeBuffer[]` pattern to `writeInt` and `writeShort`:
   
   ```java
   public final void writeInt(int v) throws IOException {
     writeBuffer[0] = (byte) (v >>> 0);
     writeBuffer[1] = (byte) (v >>> 8);
     writeBuffer[2] = (byte) (v >>> 16);
     writeBuffer[3] = (byte) (v >>> 24);
     out.write(writeBuffer, 0, 4);
   }
   ```
   
   This collapses 4 `write(int)` calls into 1 `write(byte[], int, int)` call, cutting the bookkeeping overhead by ~4x per int. Matches the existing `writeLong` pattern in the same file.
   
   Resolves the existing `TODO` in the source.
   
   ## Expected impact
   
   Standalone JMH benchmark of the class:
   - `IntEncodingBenchmark.encodePlain` (when routed through `LittleEndianDataOutputStream`): **~+35%** (~20.9M → ~28.2M ops/s)
   
   ## Note on context
   
   PR #3496 deprecates `LittleEndianDataOutputStream` because Parquet's own writers no longer use it (they write directly into `ByteBuffer`-backed slabs, which compiles to a single intrinsic store on little-endian architectures and is strictly faster than any wrapper).
   
   This PR is therefore a **purely external-caller benefit**: any third-party Parquet-format producer still using the class will get the speedup until they migrate. The change is minimal (~10 lines), obviously correct (matches the existing `writeLong` pattern), and resolves a long-standing `TODO` in the source.
   
   ## Files affected
   
   - `parquet-common/src/main/java/org/apache/parquet/bytes/LittleEndianDataOutputStream.java`
   
   No public API change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Optimize LittleEndianDataOutputStream.writeInt/writeShort with bulk write (resolves existing TODO) [parquet-java]

Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia commented on issue #3518:
URL: https://github.com/apache/parquet-java/issues/3518#issuecomment-4290348656

   Folded into #3495 / PR #3496.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [I] Optimize LittleEndianDataOutputStream.writeInt/writeShort with bulk write (resolves existing TODO) [parquet-java]

Posted by "iemejia (via GitHub)" <gi...@apache.org>.
iemejia closed issue #3518: Optimize LittleEndianDataOutputStream.writeInt/writeShort with bulk write (resolves existing TODO)
URL: https://github.com/apache/parquet-java/issues/3518


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org