You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@parquet.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/05/11 22:17:25 UTC

[PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

yadavay-amzn opened a new pull request, #3556:
URL: https://github.com/apache/parquet-java/pull/3556

   ## Problem
   
   `FallbackValuesWriter` calls `isCompressionSatisfying()` after the first page to decide whether dictionary encoding is worthwhile. With modern page-index defaults (~20k rows per page), this check fires too early for moderate-cardinality columns — dictionary encoding gets abandoned before enough data has accumulated to show its benefit, resulting in significantly larger files.
   
   As reported in #3479, a column with 1M int64 values mod 32768 produces 8.4MB with the premature fallback vs 2.2MB when dictionary encoding is preserved.
   
   ## Fix
   
   Add a configurable property `ParquetProperties.isDictionaryEarlyCheckEnabled()` (default: `true` for backward compatibility) that controls whether the first-page compression check is performed in `FallbackValuesWriter.getBytes()`.
   
   When disabled, dictionary encoding is only abandoned when the dictionary itself exceeds size limits (`shouldFallBack()`), not based on the first-page compression ratio.
   
   ## Changes
   
   - `ParquetProperties`: added `dictionaryEarlyCheckEnabled` field, getter, and builder method
   - `FallbackValuesWriter`: added overloaded `of()` factory and constructor accepting the flag; guarded the `isCompressionSatisfying` call
   - `DefaultValuesWriterFactory`: passes the config through to `FallbackValuesWriter.of()`
   - New test `TestFallbackValuesWriter`: verifies dictionary encoding is preserved when the check is disabled
   
   ## Testing
   
   - New unit tests pass (2/2)
   - Existing `parquet-column` tests unaffected (default `true` preserves existing behavior)
   
   ## Generative AI
   
   Generated-by: Claude Opus 4.7
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3376665956


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +89,9 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    cumulativeRawBytes += rawDataByteSize;

Review Comment:
   Practically i dont think we'll hit overflow but i added a check just in case



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3376599475


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -30,7 +30,12 @@ public class FallbackValuesWriter<I extends ValuesWriter & RequiresFallback, F e
 
   public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
       I initialWriter, F fallBackWriter) {
-    return new FallbackValuesWriter<>(initialWriter, fallBackWriter);
+    return new FallbackValuesWriter<>(initialWriter, fallBackWriter, /*checkAfterBytes=*/ 0);

Review Comment:
   Renamed `checkAfterBytes` to `dictionaryCheckThresholdRawSizeBytes` throughout



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4619132951

   Thanks for taking a look @wgtmac !
   
   I like the idea of having a threshold on the raw data to delay the compression check. Updated the check in `FallbackValuesWriter.getBytes()` to only fire if rawDataByteSize >= threshold (defaulting to 1MB). 
   
   I think the page count approach (checking after N pages) is sensitive to page size configurations. So with 1KB pages you'd need many pages to accumulate meaningful data, while with 1MB pages a single page might be enough. A byte threshold adapts naturally regardless of page size settings.
   
   That being said I'd like to know which approach you'd prefer and iterate based on it. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4654017073

   @wgtmac Fixed the nits. Thanks again!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "gszadovszky (via GitHub)" <gi...@apache.org>.
gszadovszky commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4629254035

   I don't think that integration tests validating the result files metadata should be a concern. Parquet-java should have the freedom to change it's default behavior related to encoding/page limits etc. until according to the specification and the configured properties. These are not breaking changes.
   
   I think we should make our changes as automatic and configuration free as possible. Introducing new configuration with staying with the current default behavior would lead to nobody actually using it. I'm OK with having the default value so the default behavior is unchanged until we want to benchmark the system in prod so we can come up with better defaults.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4697120061

   Good catch, thanks @wgtmac. You'''re right — the trigger became cumulative but the comparison was still per-page against the cumulative dictionary, which biases toward fallback the later the check fires.
   
   I'''ve pushed a fix that tracks `cumulativeEncodedBytes` alongside `cumulativeRawBytes` and changes the decision to `isCompressionSatisfying(cumulativeRawBytes, cumulativeEncodedBytes)`, so the column-chunk dictionary is amortized over all the pages it covers rather than charged against a single page. The encoded size is accumulated from the `BytesInput` already produced each page (no extra `getBytes()` calls), and both accumulators now use `Math.addExact` consistently and reset together in `resetDictionary()`.
   
   I added a test (`TestFallbackCumulativeBias`) that reproduces exactly the case you described — repeated/moderate-cardinality values across multiple pages with a finite threshold that fires on a later page. The arithmetic witness:
   - **Before (per-page):** `encoded(93) + dict(400) = 493 >= pageRaw(400)` -> fallback
   - **After (cumulative):** `totalEncoded(173) + dict(400) = 573 < totalRaw(800)` -> dictionary kept
   
   It fails on the previous commit and passes now. I also added an adversarial test confirming a genuinely high-cardinality column still correctly falls back (the fix amortizes the dictionary cost, it doesn'''t disable fallback), plus a `resetDictionary` isolation test so one chunk'''s history doesn'''t leak into the next.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3360665791


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -57,16 +70,16 @@ public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter
    */
   private long rawDataByteSize = 0;
 
-  /**
-   * indicates if this is the first page being processed
-   */
-  private boolean firstPage = true;
-
   public FallbackValuesWriter(I initialWriter, F fallBackWriter) {
+    this(initialWriter, fallBackWriter, 0);

Review Comment:
   ```suggestion
       this(initialWriter, fallBackWriter, /*checkAfterBytes=*/0);
   ```



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -44,6 +49,14 @@ public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter
 
   private boolean fellBackAlready = false;
 
+  private boolean compressionChecked = false;
+
+  private final long checkAfterBytes;
+
+  /* Accumulates raw bytes across pages (only reset in resetDictionary) so the
+   * threshold check works even when individual pages are smaller than checkAfterBytes. */
+  private long cumulativeRawBytes = 0;

Review Comment:
   ```suggestion
     private boolean compressionChecked = false;
     private final long checkAfterBytes;
     /* Accumulates raw bytes across pages (only reset in resetDictionary) so the
      * threshold check works even when individual pages are smaller than checkAfterBytes. */
     private long cumulativeRawBytes = 0;
   ```
   
   How about removing these blank lines?



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -30,7 +30,12 @@ public class FallbackValuesWriter<I extends ValuesWriter & RequiresFallback, F e
 
   public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
       I initialWriter, F fallBackWriter) {
-    return new FallbackValuesWriter<>(initialWriter, fallBackWriter);
+    return new FallbackValuesWriter<>(initialWriter, fallBackWriter, 0);

Review Comment:
   ```suggestion
       return new FallbackValuesWriter<>(initialWriter, fallBackWriter, /*checkAfterBytes=*/0);
   ```



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -167,6 +180,7 @@ private void fallBack() {
   @Override
   public void writeByte(int value) {
     rawDataByteSize += 1;
+    cumulativeRawBytes += 1;

Review Comment:
   Should we use `cumulativeRawBytes + rawDataByteSize` for checking and only increase `cumulativeRawBytes` once a page to save some cycles?



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java:
##########
@@ -161,6 +161,7 @@ public static enum JobSummaryLevel {
   public static final String BLOCK_ROW_COUNT_LIMIT = "parquet.block.row.count.limit";
   public static final String PAGE_ROW_COUNT_LIMIT = "parquet.page.row.count.limit";
   public static final String PAGE_WRITE_CHECKSUM_ENABLED = "parquet.page.write-checksum.enabled";
+  public static final String DICTIONARY_CHECK_AFTER_BYTES = "parquet.dictionary.check.after.raw.bytes";

Review Comment:
   TBH, the flag name still looks a little bit unclear to me. How about renaming it to `parquet.dictionary.check.threshold.raw.size.bytes` and also change associated variable and function names?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-5649564009

   Closing this pull request due to at least 3 months of inactivity. If you would like to continue the work, please feel free to reopen this pull request or open a new one. Thank you for your contribution to Apache Parquet!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3378291523


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +91,11 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    if (cumulativeRawBytes + rawDataByteSize >= 0) {
+      cumulativeRawBytes += rawDataByteSize;

Review Comment:
   Why not calling `Math.addExact` just like others?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4634123313

   @wgtmac Addressed all comments:
   
   - Renamed config to `parquet.dictionary.check.threshold.raw.size.bytes`
   - Performance: accumulate `cumulativeRawBytes` once per page (in `getBytes()`) instead of per value
   - Added multi-page test proving threshold is crossed after `reset()`
   - Javadoc fixed ("0 = check on first page")
   - Negative value validation added
   - Applied code suggestions (named arg comments, field grouping, Javadoc on cumulativeRawBytes)
   - Rebased on master (clean 7-file diff now)
   
   Thanks for the thorough review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3556: PARQUET-3479: Add configuration to disable early dictionary compression check
URL: https://github.com/apache/parquet-java/pull/3556


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3370723460


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +89,9 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    cumulativeRawBytes += rawDataByteSize;

Review Comment:
   Do we need to check overflow just in case?



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java:
##########
@@ -771,6 +771,17 @@ public SELF withPageWriteChecksumEnabled(boolean enablePageWriteChecksum) {
       return self();
     }
 
+    /**
+     * Set the raw data byte threshold after which the dictionary compression check is performed.
+     *
+     * @param val byte threshold (0 means check on the first page / preserve previous behavior)

Review Comment:
   ```suggestion
        * @param val byte threshold (0 means checking on the first page for every column chunk)
   ```



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -30,7 +30,12 @@ public class FallbackValuesWriter<I extends ValuesWriter & RequiresFallback, F e
 
   public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
       I initialWriter, F fallBackWriter) {
-    return new FallbackValuesWriter<>(initialWriter, fallBackWriter);
+    return new FallbackValuesWriter<>(initialWriter, fallBackWriter, /*checkAfterBytes=*/ 0);
+  }
+
+  public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
+      I initialWriter, F fallBackWriter, long checkAfterBytes) {

Review Comment:
   ditto and same for all below



##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -322,6 +325,17 @@ public boolean getPageWriteChecksumEnabled() {
     return pageWriteChecksumEnabled;
   }
 
+  /**
+   * Returns the byte threshold after which the dictionary compression check is performed.
+   * A value of 0 means check on the first page (backward compatible default). Higher values

Review Comment:
   IMO, we don't need to mention `backward compatible default` everywhere.



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -30,7 +30,12 @@ public class FallbackValuesWriter<I extends ValuesWriter & RequiresFallback, F e
 
   public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
       I initialWriter, F fallBackWriter) {
-    return new FallbackValuesWriter<>(initialWriter, fallBackWriter);
+    return new FallbackValuesWriter<>(initialWriter, fallBackWriter, /*checkAfterBytes=*/ 0);

Review Comment:
   `checkAfterBytes` now should be replaced by a more consistent name with `dictionaryCheckThresholdRawSizeBytes`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3376601056


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -30,7 +30,12 @@ public class FallbackValuesWriter<I extends ValuesWriter & RequiresFallback, F e
 
   public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
       I initialWriter, F fallBackWriter) {
-    return new FallbackValuesWriter<>(initialWriter, fallBackWriter);
+    return new FallbackValuesWriter<>(initialWriter, fallBackWriter, /*checkAfterBytes=*/ 0);
+  }
+
+  public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter> FallbackValuesWriter<I, F> of(
+      I initialWriter, F fallBackWriter, long checkAfterBytes) {

Review Comment:
   Renamed `checkAfterBytes` to `dictionaryCheckThresholdRawSizeBytes` throughout



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4626286799

   @wgtmac Good catches, thanks. Updated:
   
   1. **Renamed** config to `parquet.dictionary.check.after.raw.bytes` to clarify it refers to uncompressed bytes.
   
   2. **Fixed the reset issue.** You were right — `rawDataByteSize` resets per page via `reset()`, so it would never reach a 1MB threshold. Added a separate `cumulativeRawBytes` counter that accumulates across pages and only resets in `resetDictionary()` (between column chunks). The threshold gate uses `cumulativeRawBytes`; the actual compression comparison still uses the current page's `rawDataByteSize` vs encoded size — same apples-to-apples comparison as before.
   
   3. **Default is now 0** (backward compatible — check fires on first page, same as old `firstPage` behavior). Users can opt in to a higher threshold to delay the check.
   
   4. **Null-heavy columns:** with default 0 the check fires on the first page regardless of null ratio, same as before. With a higher threshold, nulls don't contribute to `cumulativeRawBytes` (they're in definition levels), so the threshold takes longer to reach — but the check still eventually fires once enough non-null values accumulate. For all-null columns the check fires immediately since `cumulativeRawBytes >= 0` is trivially true.
   
   Let me know if this direction works or if you'd prefer a different approach.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-5274278638

   This pull request has been automatically marked as stale because it has had no activity for at least 2 months. If you are still working on this change or plan to move it forward, please leave a comment or push a new commit so we know to keep it open. Otherwise, this PR will be closed automatically in about one month. Thank you for your contribution to Apache Parquet!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3376666710


##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -322,6 +325,17 @@ public boolean getPageWriteChecksumEnabled() {
     return pageWriteChecksumEnabled;
   }
 
+  /**
+   * Returns the byte threshold after which the dictionary compression check is performed.
+   * A value of 0 means check on the first page (backward compatible default). Higher values

Review Comment:
   Removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4609512333

   Thanks for looking into this! 
   
   While the problem is real, introducing a boolean flag `dictionaryEarlyCheckEnabled` feels like a band-aid fix that pushes the burden to users. Most users won't know when to manually toggle this to prevent storage inflation.
   
   Instead of a new config, could we make the heuristic more adaptive? For example, we could delay the compression check until we've accumulated a certain amount of raw data (e.g., `1MB`), or evaluate it over the first `N` pages rather than just the first one. 
   
   This would solve the issue out of the box without hurting usability. Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4619402500

   Thanks for the quick update. Before diving into the code detail, I have some additional questions:
   
   - `parquet.dictionary.check.after.bytes` is confusing. Does "bytes" refer to raw (uncompressed) bytes or compressed/encoded bytes? Please rename it to be explicit (e.g., parquet.dictionary.check.after.raw.bytes) and update the documentation accordingly.
   - In FallbackValuesWriter, rawDataByteSize is reset to 0 at the end of every page via reset(). With modern configurations (e.g., 20k rows per page), a single page's size will almost never reach the 1MB threshold. This effectively disables the compression check entirely. Did you intend for this data size to accumulate across pages, or is permanently bypassing the check for small pages your actual design?
   - Since nulls are encoded in definition levels and do not contribute to rawDataByteSize, a page with a high null ratio will have an extremely small raw data size. How does this algorithm account for heavily null-populated columns? Will it also indefinitely bypass the check?
   - It would be good to make the default behavior unchanged since many downstream environments (e.g. Apache Spark) unfortunately check produced file metadata (including encoding, file size, etc.) in their integration tests so behavior change may break them.
   
   cc @gszadovszky @Fokko for additional advice.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3394400538


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +91,13 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    try {
+      cumulativeRawBytes = Math.addExact(cumulativeRawBytes, rawDataByteSize);
+    } catch (ArithmeticException e) {
+      // overflow, keep the previous value
+    }
+    if (!fellBackAlready && !compressionChecked && cumulativeRawBytes >= dictionaryCheckThresholdRawSizeBytes) {
+      compressionChecked = true;
       BytesInput bytes = initialWriter.getBytes();
       if (!initialWriter.isCompressionSatisfying(rawDataByteSize, bytes.size())) {

Review Comment:
   I think the threshold gate now accumulates raw bytes across pages, but the compression check still compares the current page raw size against the current page encoded size plus the whole accumulated dictionary size. This seems biased toward fallback once the check is delayed beyond the first page. Should this comparison use cumulative raw/encoded sizes, or otherwise avoid charging the full column-chunk dictionary size to a single page? A finite-threshold test with repeated/moderate-cardinality values across pages would help verify the intended PARQUET-3479 case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3360643554


##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetOutputFormat.java:
##########
@@ -161,6 +161,7 @@ public static enum JobSummaryLevel {
   public static final String BLOCK_ROW_COUNT_LIMIT = "parquet.block.row.count.limit";
   public static final String PAGE_ROW_COUNT_LIMIT = "parquet.page.row.count.limit";
   public static final String PAGE_WRITE_CHECKSUM_ENABLED = "parquet.page.write-checksum.enabled";
+  public static final String DICTIONARY_CHECK_AFTER_BYTES = "parquet.dictionary.check.after.raw.bytes";

Review Comment:
   Could we also document `parquet.dictionary.check.after.raw.bytes` in the configuration list above? It would be useful to mention that this is based on raw value bytes, and nulls encoded in definition levels do not contribute to this threshold.



##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -709,6 +725,19 @@ public Builder withPageWriteChecksumEnabled(boolean val) {
       return this;
     }
 
+    /**
+     * Set the raw data byte threshold after which the dictionary compression check is performed.
+     * A value of 0 means check on the first page (backward compatible default). Higher values
+     * delay the check until that many raw bytes have been accumulated across pages.
+     *
+     * @param val byte threshold (default: 0)
+     * @return this builder for method chaining
+     */
+    public Builder withDictionaryCheckAfterBytes(long val) {
+      this.dictionaryCheckAfterBytes = val;

Review Comment:
   Should we reject negative values here? A negative threshold effectively behaves like `0`, but accepting it silently seems a bit confusing for a size-like config. Most nearby size/count options validate the input, so `val >= 0` would be clearer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3360641976


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +92,8 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    if (!fellBackAlready && !compressionChecked && cumulativeRawBytes >= checkAfterBytes) {

Review Comment:
   Could we add a test where the threshold is crossed only after a `reset()`? This is the exact case we want to protect here, since `rawDataByteSize` is reset per page while `cumulativeRawBytes` should keep accumulating across pages. The current tests with `0` and `Long.MAX_VALUE` would not catch a regression back to per-page counting.



##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetWriter.java:
##########
@@ -771,6 +771,17 @@ public SELF withPageWriteChecksumEnabled(boolean enablePageWriteChecksum) {
       return self();
     }
 
+    /**
+     * Set the raw data byte threshold after which the dictionary compression check is performed.
+     *
+     * @param val byte threshold (0 means check on every page)

Review Comment:
   Nit: this says `0 means check on every page`, but the implementation only checks once per column chunk, same as the old first-page behavior. Could we update this to say `0` means check on the first page / preserve the previous behavior?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3378132635


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +91,11 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    if (cumulativeRawBytes + rawDataByteSize >= 0) {
+      cumulativeRawBytes += rawDataByteSize;
+    }

Review Comment:
   `cumulativeRawBytes + rawDataByteSize >= 0` is effectively always true for this writer (rawDataByteSize is only incremented). The conditional adds an unnecessary branch and contradicts the nearby comment stating overflow is not a concern. If overflow handling is desired, it should be done explicitly (e.g., saturating add); otherwise, just accumulate directly.



##########
parquet-column/src/main/java/org/apache/parquet/column/ParquetProperties.java:
##########
@@ -322,6 +325,17 @@ public boolean getPageWriteChecksumEnabled() {
     return pageWriteChecksumEnabled;
   }
 
+  /**
+   * Returns the byte threshold after which the dictionary compression check is performed.
+   * A value of 0 means check on the first page. Higher values delay the check until that
+   * many raw bytes have been accumulated across pages.
+   *
+   * @return the byte threshold for the dictionary compression check
+   */
+  public long getDictionaryCheckThresholdRawSizeBytes() {
+    return dictionaryCheckThresholdRawSizeBytes;
+  }

Review Comment:
   PR description mentions adding a boolean `isDictionaryEarlyCheckEnabled()` to disable the early compression check, but this change introduces a byte-threshold (`getDictionaryCheckThresholdRawSizeBytes`) instead. To avoid confusion for reviewers and users, please update the PR description (or related docs/release notes) to match the threshold-based configuration that was actually implemented.



##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -57,16 +69,16 @@ public static <I extends ValuesWriter & RequiresFallback, F extends ValuesWriter
    */
   private long rawDataByteSize = 0;
 
-  /**
-   * indicates if this is the first page being processed
-   */
-  private boolean firstPage = true;
-
   public FallbackValuesWriter(I initialWriter, F fallBackWriter) {
+    this(initialWriter, fallBackWriter, /*dictionaryCheckThresholdRawSizeBytes=*/ 0);
+  }
+
+  public FallbackValuesWriter(I initialWriter, F fallBackWriter, long dictionaryCheckThresholdRawSizeBytes) {
     super();
     this.initialWriter = initialWriter;
     this.fallBackWriter = fallBackWriter;
     this.currentWriter = initialWriter;
+    this.dictionaryCheckThresholdRawSizeBytes = dictionaryCheckThresholdRawSizeBytes;
   }

Review Comment:
   The new `dictionaryCheckThresholdRawSizeBytes` parameter is not validated in this public constructor/factory path. `ParquetProperties.Builder` enforces `>= 0`, but callers can still create `FallbackValuesWriter` directly with a negative value, which would make the compression check run immediately and hide configuration errors. Consider rejecting negative values here as well for robustness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#issuecomment-4608357572

   @Fokko Could you take a look? This adds a config (`parquet.dictionary.early.check.enabled`) to disable the first-page compression check in FallbackValuesWriter. With modern page-index defaults (~20k rows/page), the check fires too early for moderate-cardinality columns, abandoning dictionary encoding prematurely. Includes unit test + E2E integration test writing real Parquet files. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org


Re: [PR] PARQUET-3479: Add configuration to disable early dictionary compression check [parquet-java]

Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on code in PR #3556:
URL: https://github.com/apache/parquet-java/pull/3556#discussion_r3407173911


##########
parquet-column/src/main/java/org/apache/parquet/column/values/fallback/FallbackValuesWriter.java:
##########
@@ -79,8 +91,11 @@ public long getBufferedSize() {
 
   @Override
   public BytesInput getBytes() {
-    if (!fellBackAlready && firstPage) {
-      // we use the first page to decide if we're going to use this encoding
+    if (cumulativeRawBytes + rawDataByteSize >= 0) {
+      cumulativeRawBytes += rawDataByteSize;

Review Comment:
   fixed this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@parquet.apache.org
For additional commands, e-mail: issues-help@parquet.apache.org