You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org> on 2026/08/01 05:50:21 UTC

[PR] Parquet: Fix per-column statistics disabling across multiple columns [iceberg]

Alwaysgaurav1 opened a new pull request, #17461:
URL: https://github.com/apache/iceberg/pull/17461

   Closes #15347.
   
   `Parquet.setColumnStatsConfig` previously applied per-column statistics enablement via `withColumnStatsEnabled.accept(...)` on `ParquetProperties.Builder` / `ParquetWriter.Builder` but did not propagate `parquet.column.statistics.enabled#<column>` into the Hadoop `Configuration` instance (`conf`). 
   
   When users configured multiple table properties to disable statistics across multiple columns (e.g., `write.parquet.stats-enabled.column.colA = false` and `write.parquet.stats-enabled.column.colB = false`), statistics were only disabled for the first column processed while subsequent columns continued to write statistics into the Parquet file header.
   
   This change explicitly passes per-column statistics enablement settings into `conf` as well (`parquet.column.statistics.enabled#<parquetColumnPath>`), ensuring consistent behavior across all configured columns regardless of write path.
   
   ## Tests Added
   - `testMultipleColumnStatisticsDisabled` in `TestParquet.java`: Validates that configuring `write.parquet.stats-enabled.column.<col> = false` across multiple columns disables statistics for all specified columns in the generated Parquet file.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Fix per-column statistics disabling across multiple columns [iceberg]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17461:
URL: https://github.com/apache/iceberg/pull/17461#discussion_r3700434548


##########
parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java:
##########
@@ -363,7 +363,11 @@ private void setColumnStatsConfig(
                   LOG.warn("Skipping column statistics config for missing field: {}", colPath);
                   return;
                 }
-                withColumnStatsEnabled.accept(parquetColumnPath, Boolean.valueOf(isEnabled));
+                boolean enabled = Boolean.parseBoolean(isEnabled);
+                withColumnStatsEnabled.accept(parquetColumnPath, enabled);
+                conf.set(
+                    "parquet.column.statistics.enabled#" + parquetColumnPath,

Review Comment:
   The added conf.set("parquet.column.statistics.enabled#" + parquetColumnPath, String.valueOf(enabled)) is dead code. The key parquet.column.statistics.enabled#<col> is consumed only by ParquetOutputFormat.ColumnConfigParser (the Hadoop MapReduce write path); Iceberg never invokes ParquetOutputFormat, so this key is never read back by either the createWriterFunc path (ParquetProperties.Builder, line 487) or the ParquetWriteBuilder path (line 532). The real per-column setting routes through withColumnStatsEnabled.accept(parquetColumnPath, enabled) at line 367 → propsBuilder::withStatisticsEnabled / parquetWriteBuilder::withStatisticsEnabled → ColumnProperty.Builder HashMap, which already handles multiple columns independently. The production change is based on a false root-cause diagnosis and should be reverted entirely.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Fix per-column statistics disabling across multiple columns [iceberg]

Posted by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org>.
Alwaysgaurav1 closed pull request #17461: Parquet: Fix per-column statistics disabling across multiple columns
URL: https://github.com/apache/iceberg/pull/17461


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Fix per-column statistics disabling across multiple columns [iceberg]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #17461:
URL: https://github.com/apache/iceberg/pull/17461#discussion_r3700435479


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -288,6 +288,55 @@ public void testColumnStatisticsEnabled() throws Exception {
     }
   }
 
+  @Test
+  public void testMultipleColumnStatisticsDisabled() throws Exception {
+    Schema schema =
+        new Schema(
+            optional(1, "int_field", IntegerType.get()),
+            optional(2, "string_field", Types.StringType.get()),
+            optional(3, "long_field", Types.LongType.get()));
+
+    File file = createTempFile(temp);
+
+    List<GenericData.Record> records = Lists.newArrayListWithCapacity(5);
+    org.apache.avro.Schema avroSchema = AvroSchemaUtil.convert(schema.asStruct());
+    for (int i = 1; i <= 5; i++) {
+      GenericData.Record record = new GenericData.Record(avroSchema);
+      record.put("int_field", i);
+      record.put("string_field", "test");
+      record.put("long_field", (long) i);
+      records.add(record);
+    }
+
+    write(
+        file,
+        schema,
+        ImmutableMap.<String, String>builder()
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "int_field", "false")
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "string_field", "false")
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "long_field", "true")
+            .buildOrThrow(),
+        ParquetAvroWriter::buildWriter,
+        records.toArray(new GenericData.Record[] {}));
+
+    InputFile inputFile = Files.localInput(file);
+
+    try (ParquetFileReader reader = ParquetFileReader.open(ParquetIO.file(inputFile))) {
+      for (BlockMetaData block : reader.getFooter().getBlocks()) {
+        for (ColumnChunkMetaData column : block.getColumns()) {
+          boolean emptyStats = column.getStatistics().isEmpty();
+          if (column.getPath().toDotString().equals("int_field")) {
+            assertThat(emptyStats).as("int_field has statistics disabled").isEqualTo(true);
+          } else if (column.getPath().toDotString().equals("string_field")) {
+            assertThat(emptyStats).as("string_field has statistics disabled").isEqualTo(true);
+          } else if (column.getPath().toDotString().equals("long_field")) {
+            assertThat(emptyStats).as("long_field has statistics enabled").isEqualTo(false);
+          }
+        }
+      }
+    }
+  }
+

Review Comment:
   No production fix is needed: the multi-column statistics disable path already works on current main. PR #17365 (open, approved) adds equivalent test coverage without any production change and CI is green, confirming the multi-column path is not broken. This PR's test testMultipleColumnStatisticsDisabled does not guard a real regression; it passes with or without the conf.set because the actual enforcement runs through ParquetProperties.Builder, not conf.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet: Fix per-column statistics disabling across multiple columns [iceberg]

Posted by "Alwaysgaurav1 (via GitHub)" <gi...@apache.org>.
Alwaysgaurav1 commented on code in PR #17461:
URL: https://github.com/apache/iceberg/pull/17461#discussion_r3702371128


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -288,6 +288,55 @@ public void testColumnStatisticsEnabled() throws Exception {
     }
   }
 
+  @Test
+  public void testMultipleColumnStatisticsDisabled() throws Exception {
+    Schema schema =
+        new Schema(
+            optional(1, "int_field", IntegerType.get()),
+            optional(2, "string_field", Types.StringType.get()),
+            optional(3, "long_field", Types.LongType.get()));
+
+    File file = createTempFile(temp);
+
+    List<GenericData.Record> records = Lists.newArrayListWithCapacity(5);
+    org.apache.avro.Schema avroSchema = AvroSchemaUtil.convert(schema.asStruct());
+    for (int i = 1; i <= 5; i++) {
+      GenericData.Record record = new GenericData.Record(avroSchema);
+      record.put("int_field", i);
+      record.put("string_field", "test");
+      record.put("long_field", (long) i);
+      records.add(record);
+    }
+
+    write(
+        file,
+        schema,
+        ImmutableMap.<String, String>builder()
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "int_field", "false")
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "string_field", "false")
+            .put(PARQUET_COLUMN_STATS_ENABLED_PREFIX + "long_field", "true")
+            .buildOrThrow(),
+        ParquetAvroWriter::buildWriter,
+        records.toArray(new GenericData.Record[] {}));
+
+    InputFile inputFile = Files.localInput(file);
+
+    try (ParquetFileReader reader = ParquetFileReader.open(ParquetIO.file(inputFile))) {
+      for (BlockMetaData block : reader.getFooter().getBlocks()) {
+        for (ColumnChunkMetaData column : block.getColumns()) {
+          boolean emptyStats = column.getStatistics().isEmpty();
+          if (column.getPath().toDotString().equals("int_field")) {
+            assertThat(emptyStats).as("int_field has statistics disabled").isEqualTo(true);
+          } else if (column.getPath().toDotString().equals("string_field")) {
+            assertThat(emptyStats).as("string_field has statistics disabled").isEqualTo(true);
+          } else if (column.getPath().toDotString().equals("long_field")) {
+            assertThat(emptyStats).as("long_field has statistics enabled").isEqualTo(false);
+          }
+        }
+      }
+    }
+  }
+

Review Comment:
   "Thanks @uros-b for the review and explanation! Makes total sense — closing this PR in favor of #17365."



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org