You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "arouel (via GitHub)" <gi...@apache.org> on 2026/04/19 17:32:12 UTC

[I] Avoid per-write virtual dispatch in `DictionaryValuesWriter.shouldFallBack()` by caching the size-exceeded check [parquet-java]

arouel opened a new issue, #3501:
URL: https://github.com/apache/parquet-java/issues/3501

   ### Describe the enhancement requested
   
   `DictionaryValuesWriter.shouldFallBack()` is called by `FallbackValuesWriter.checkFallback()` after every single value write. The current implementation dispatches a virtual call to `getDictionarySize()` on every invocation:
   ```java
   public boolean shouldFallBack() {
     return dictionaryByteSize > maxDictionaryByteSize || getDictionarySize() > MAX_DICTIONARY_ENTRIES;
   }
   ```
   `getDictionarySize()` is an abstract method overridden in each typed subclass (Binary, Long, Double, Integer, Float) to return the backing map's `.size()`. Since `shouldFallBack()` is polled after every write, including writes of duplicate values that do not grow the dictionary, the virtual dispatch and map-size query are redundant work for the common case where most values are already in the dictionary.
   Both `dictionaryByteSize` and the dictionary entry count can only increase when a new entry is added (inside the `if (id == -1)` branch of each subclass's write method). Therefore the size-exceeded condition can only transition from `false` to `true` at that exact point.
   
   ### Proposal
   Replace the per-write check with a cached boolean `dictionarySizeExceeded` flag. Introduce a `checkDictionarySizeLimit(int newDictionarySize)` method that subclass write methods call only when a new dictionary entry is actually added. `shouldFallBack()` then returns the cached flag directly, a simple field read with no virtual dispatch.
   
   ### Component(s)
   
   Core


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org