You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nssalian (via GitHub)" <gi...@apache.org> on 2026/07/29 22:26:35 UTC

[PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

nssalian opened a new pull request, #17424:
URL: https://github.com/apache/iceberg/pull/17424

   ## Rationale for the change
   
   The current majority-based algorithm introduced in [#14297](https://github.com/apache/iceberg/pull/14297) shreds mixed-type fields to the majority type. A field seen as 95% int + 5% string still emits an int-typed column that misses on the 5% string rows, which fall back to the residual `value`. That `typed_value` column pays its storage and I/O cost while covering only a fraction of the field's rows, and readers must always check both columns.
   
   This change requires type uniformity: a field is shredded only when all its observations fall into a single type family after numeric widening. Mixed-type fields stay in the residual `value`; no `typed_value` column is emitted for them.
   
   ### Benefits
   
   - Reader I/O: no mostly-empty typed_value columns to open, decode, and skip. Wide-schema files get fewer column chunks and less Parquet metadata.
   - Predicate pushdown and row-group skipping: when a typed_value column exists, its min/max stats cover the entire field, so filter pushdown and stats-based row-group skipping are clean. Under the majority algorithm, minority-type rows in `value` broke stat coverage and forced fallback scans.
   - Predictable rule: uniform -> shredded, otherwise -> residual. No surprising tie-break outcomes (e.g., 50/50 int-vs-string silently picking STRING via priority table).
   
   
   Clean single-type workloads are unchanged. Benchmark evidence (per-column scoring plus reader wall-clock across 11 workloads at 100k rows) shows uniformity ties or beats the majority algorithm on every workload and wins uniquely on mixed-type inputs (60/40, 95/5, 99/1 int/string, four-way polymorphic). Full numbers in the design document.
   
   Design document with benchmarks: [Google Doc](https://docs.google.com/document/d/1OaB45mg9JJ_JRg-ipuvyh4AtOmaGmdWMsy_mJKiu5xs/edit?usp=sharing)
   
   ## Changes
   
   - `VariantShreddingAnalyzer`: add a uniformity check to `admittedType()`. Returns null when observations span more than one type family after widening; null propagates through existing handling in `analyzeAndCreateSchema`, `buildFieldGroup`, and `createArrayTypedValue`, so rejected fields fall out of the emitted schema and stay in residual `value`.
   - Removed the now-unreachable `TIE_BREAK_PRIORITY` map; simplified type selection to `families.iterator().next()`.
   - Rename `getMostCommonType` -> `admittedType` (plus `mostCommonCached`/`Computed` and local renames). Change `combinedCounts` from `Map<PhysicalType, Integer>` to `Set<PhysicalType> families` since counts are no longer needed post-uniformity.
   - Updated the class-level javadoc.
   
   ## Follow up
   - Will open follow-up issues for each implementation; the iceberg-go change I'll  implement myself.
   
   ## Tests
   
   - `TestVariantShreddingAnalyzer`: 5 new tests covering mixed-primitive rejection, integer widening admission, int+decimal cross-family rejection, mixed object+primitive rejection, and array-element mixing rejection. One existing test updated to use uniform-type observations.
   - `TestVariantShredding` (Spark v4.0 and v4.1): 3 tests renamed and updated to assert non-shredding on mixed inputs (`testInconsistentType`, `testPrimitiveDecimalType`, `testMixedTypeTieBreaking`).
   - `TestFlinkVariantShreddingType` (Flink v2.1): 4 tests renamed and updated similarly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679137492


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric
+ *       widening; mixed-type fields remain in the residual {@code value}.
  *   <li>Integer types (INT8/16/32/64) and decimal types (DECIMAL4/8/16) are each promoted to the
- *       widest observed before competing with other types.
+ *       widest observed within their family.
  *   <li>Fields below {@code MIN_FIELD_FREQUENCY} are pruned. Above {@code MAX_SHREDDED_FIELDS}, the

Review Comment:
   This probably got answered when this was originally written, but why aren't floats in the same family?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679219293


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I generally like to avoid having multiple local variables that only work if they are set in certain combinations and I think we can make this a bit tighter if instead of tracking all these things separately we do something like.
   
   if admitted not set - set
   if admtted set and this type is wider, set admitted to wider type
   if admitted set and this is not  in the family - return null exit early on everything
   
   here is a quick draft i had the llm do
   
   ```java
   PhysicalType admittedType() {
     PhysicalType admitted = null;
     for (int i = 0; i < typeCounts.length; i++) {
       if (typeCounts[i] == 0) {
         continue;
       }
       PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
       if (merged == null) {
         // Mixed type families: do not shred this field.
         return null;
       }
       admitted = merged;
     }
     return admitted;
   }
   /**
    * Merges {@code candidate} into the currently admitted type.
    *
    * <p>Returns the wider type when both are in the same integer or decimal family, {@code
    * candidate} when nothing is admitted yet, {@code current} when the types are identical, and
    * null when the types are from incompatible families (including FLOAT vs DOUBLE).
    */
   private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
     if (current == null) {
       return candidate;
     }
     if (current == candidate) {
       return current;
     }
     if (isIntegerType(current) && isIntegerType(candidate)) {
       return integerRank(current) >= integerRank(candidate) ? current : candidate;
     }
     if (isDecimalType(current) && isDecimalType(candidate)) {
       return decimalRank(current) >= decimalRank(candidate) ? current : candidate;
     }
     return null;
   }
   private static int integerRank(PhysicalType type) {
     return switch (type) {
       case INT8 -> 0;
       case INT16 -> 1;
       case INT32 -> 2;
       case INT64 -> 3;
       default -> throw new IllegalArgumentException("Not an integer type: " + type);
     };
   }
   private static int decimalRank(PhysicalType type) {
     return switch (type) {
       case DECIMAL4 -> 0;
       case DECIMAL8 -> 1;
       case DECIMAL16 -> 2;
       default -> throw new IllegalArgumentException("Not a decimal type: " + type);
     };
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#issuecomment-5171272961

   Thanks for the reviews @RussellSpitzer. Addressed the comments. I dropped the redundant cached/widest-type state, and the decision now only depends on which families appear, rather than counts. Added a few tests for metrics and widening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679307155


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestVariantShreddingAnalyzer.java:
##########
@@ -471,6 +469,112 @@ public void testUuidFieldIsTrackedAndShredded() {
         .isInstanceOf(LogicalTypeAnnotation.UUIDLogicalTypeAnnotation.class);
   }
 
+  @Test
+  public void testMixedPrimitiveTypesFieldNotShredded() {

Review Comment:
   I forget where we landed on this but I think new tests follow Junit 5 and don't include "test" but I guess this file already has lots of them... I forget what the community decided. Probably can stay as is



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3685773309


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric

Review Comment:
   Added a note in the javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679291386


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I would try iterating on that though ... see if we can remove all the state here we don't need



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3686359374


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +444,65 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+        admitted = merged;
       }
+      return admitted;
+    }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+    /**
+     * Merges {@code candidate} into the currently admitted type. Returns the wider type when both
+     * are in the same integer or decimal family, {@code candidate} when nothing is admitted yet,
+     * {@code current} when the types are identical, and null when the types are from incompatible
+     * families (including FLOAT vs DOUBLE, TIMESTAMPTZ vs TIMESTAMPTZ_NANOS, etc.).
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
-
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      if (current == candidate) {
+        return current;
       }
-
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
-    }
-
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+      PhysicalType widened = wider(current, candidate, INTEGER_TYPES);

Review Comment:
   may be overkill if we never add another family .. but
   
   ```java
   private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
     if (current == null) {
       return candidate;
     }
     if (current == candidate) {
       return current;
     }
     List<PhysicalType> family = familyOf(current);
     if (family == null) {
       return null; // current has no widening family (STRING, FLOAT, …)
     }
     return wider(current, candidate, family); // null if candidate not in that family
   }
   
   private static List<PhysicalType> familyOf(PhysicalType type) {
     if (INTEGER_TYPES.contains(type)) {
       return INTEGER_TYPES;
     }
     if (DECIMAL_TYPES.contains(type)) {
       return DECIMAL_TYPES;
     }
     return null;
   }
   ```
   
   I dunno ... i'll let you decide :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679345591


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   Yeah let me see if this is complete. I'll noodle on this more and update.



##########
parquet/src/test/java/org/apache/iceberg/parquet/TestVariantShreddingAnalyzer.java:
##########
@@ -471,6 +469,112 @@ public void testUuidFieldIsTrackedAndShredded() {
         .isInstanceOf(LogicalTypeAnnotation.UUIDLogicalTypeAnnotation.class);
   }
 
+  @Test
+  public void testMixedPrimitiveTypesFieldNotShredded() {
+    VariantMetadata meta = Variants.metadata("mixed", "keep");
+    ShreddedObject row1 = Variants.object(meta);
+    row1.put("mixed", Variants.of(42));
+    row1.put("keep", Variants.of(1));
+    ShreddedObject row2 = Variants.object(meta);
+    row2.put("mixed", Variants.of("text"));
+    row2.put("keep", Variants.of(2));
+
+    DirectAnalyzer analyzer = new DirectAnalyzer();
+    Type schema = analyzer.analyzeAndCreateSchema(List.of(row1, row2), 0);
+
+    assertThat(schema).isNotNull().isInstanceOf(GroupType.class);
+    GroupType typedValue = (GroupType) schema;
+    assertThat(typedValue.containsField("mixed")).isFalse();
+    assertThat(typedValue.containsField("keep")).isTrue();
+  }
+
+  @Test
+  public void testIntegerWideningAdmitsField() {

Review Comment:
   Will add



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679142695


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric
+ *       widening; mixed-type fields remain in the residual {@code value}.
  *   <li>Integer types (INT8/16/32/64) and decimal types (DECIMAL4/8/16) are each promoted to the
- *       widest observed before competing with other types.
+ *       widest observed within their family.
  *   <li>Fields below {@code MIN_FIELD_FREQUENCY} are pruned. Above {@code MAX_SHREDDED_FIELDS}, the

Review Comment:
   If we don't want to include them we probably should just have an explict mixed float/double column isn't shredded test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#issuecomment-5295724000

   Thanks @nssalian  for the pr, and thanks @Guosmilesmile for the review. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3778574938


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +446,80 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+
+        admitted = merged;
+      }
+      return admitted;
+    }
+
+    /**
+     * Widens {@code current} with {@code candidate}, or null if they belong to different families.
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+      if (current == candidate) {
+        return current;
       }
 
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      List<PhysicalType> family = familyOf(current);
+      if (family == null) {
+        return null;
       }
 
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
+      return wider(current, candidate, family);
     }
 
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+    /** Returns the widening family for {@code type}, or null if none applies. */
+    private static List<PhysicalType> familyOf(PhysicalType type) {
+      if (INTEGER_TYPES.contains(type)) {
+        return INTEGER_TYPES;
+      }
+
+      if (DECIMAL_TYPES.contains(type)) {
+        return DECIMAL_TYPES;
+      }
+
+      return null;
     }
 
-    private static boolean isDecimalType(PhysicalType type) {
-      return type == PhysicalType.DECIMAL4
-          || type == PhysicalType.DECIMAL8
-          || type == PhysicalType.DECIMAL16;
+    /**
+     * Returns the wider of {@code first} and {@code second} within {@code family}, or null when
+     * {@code second} is not in the family. {@code first} is always a member.
+     */
+    private static PhysicalType wider(

Review Comment:
   super nit: but you have familyOf and I think this would read better as widerOf



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3778886761


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +446,80 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+
+        admitted = merged;
+      }
+      return admitted;
+    }
+
+    /**
+     * Widens {@code current} with {@code candidate}, or null if they belong to different families.
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+      if (current == candidate) {
+        return current;
       }
 
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      List<PhysicalType> family = familyOf(current);
+      if (family == null) {
+        return null;
       }
 
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
+      return wider(current, candidate, family);
     }
 
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+    /** Returns the widening family for {@code type}, or null if none applies. */
+    private static List<PhysicalType> familyOf(PhysicalType type) {
+      if (INTEGER_TYPES.contains(type)) {
+        return INTEGER_TYPES;
+      }
+
+      if (DECIMAL_TYPES.contains(type)) {
+        return DECIMAL_TYPES;
+      }
+
+      return null;
     }
 
-    private static boolean isDecimalType(PhysicalType type) {
-      return type == PhysicalType.DECIMAL4
-          || type == PhysicalType.DECIMAL8
-          || type == PhysicalType.DECIMAL16;
+    /**
+     * Returns the wider of {@code first} and {@code second} within {@code family}, or null when
+     * {@code second} is not in the family. {@code first} is always a member.
+     */
+    private static PhysicalType wider(

Review Comment:
   I like it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679137492


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric
+ *       widening; mixed-type fields remain in the residual {@code value}.
  *   <li>Integer types (INT8/16/32/64) and decimal types (DECIMAL4/8/16) are each promoted to the
- *       widest observed before competing with other types.
+ *       widest observed within their family.
  *   <li>Fields below {@code MIN_FIELD_FREQUENCY} are pruned. Above {@code MAX_SHREDDED_FIELDS}, the

Review Comment:
   This probably got answered when this was originally written, but why aren't floats and doubles in the same family?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3685780184


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestVariantShreddingAnalyzer.java:
##########
@@ -471,6 +469,112 @@ public void testUuidFieldIsTrackedAndShredded() {
         .isInstanceOf(LogicalTypeAnnotation.UUIDLogicalTypeAnnotation.class);
   }
 
+  @Test
+  public void testMixedPrimitiveTypesFieldNotShredded() {

Review Comment:
   Kept the test prefix. I think newer tests have that, but a lot of tests in the codebase still have the prefix so I kept that in mind



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#issuecomment-5125425077

   Overall I think this is good. I have some style/function comments on the code itself. I really think we should drop all the state we currently have since I think at the end of the day the only state we need is 
   
   "Have we decided to shred or not?"
   "What is that shredded type?"
   
   Currently we have a widest decimal, widest integer, cached , and set 
   
   I wrote this inline but in my head this could be 
   
   Are we trying to shred to a type and haven't picked one? Set it
   
   Is the current type not in the family of the set type? We can't shred
   
   Is the current type in the family of the set type? 
      is it narrower or equal? do nothing
      is it wider? Set it
    


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679219293


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I generally like to avoid having multiple local variables that only work if they are set in certain combinations and I think we can make this a bit tighter if instead of tracking all these things separately we do something like.
   
   if admitted not set - set
   if admtted set and this type is wider, set admitted to wider type
   if admitted set and this is not  in the family - return null exit early on everything
   
   here is a quick draft i had the llm do
   
   ```java
   PhysicalType admittedType() {
     PhysicalType admitted = null;
     for (int i = 0; i < typeCounts.length; i++) {
       if (typeCounts[i] == 0) {
         continue;
       }
       PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
       if (merged == null) {
         // Mixed type families: do not shred this field.
         return null;
       }
       admitted = merged;
     }
     return admitted;
   }
   /**
    * Merges {@code candidate} into the currently admitted type.
    *
    * <p>Returns the wider type when both are in the same integer or decimal family, {@code
    * candidate} when nothing is admitted yet, {@code current} when the types are identical, and
    * null when the types are from incompatible families (including FLOAT vs DOUBLE).
    */
   private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
     if (current == null) {
       return candidate;
     }
     if (current == candidate) {
       return current;
     }
     if (isIntegerType(current) && isIntegerType(candidate)) {
       // INT8..INT64 are contiguous in PhysicalType declaration order.
       return current.ordinal() >= candidate.ordinal() ? current : candidate;
     }
     if (isDecimalType(current) && isDecimalType(candidate)) {
       // DECIMAL4..DECIMAL16 are contiguous in PhysicalType declaration order.
       return current.ordinal() >= candidate.ordinal() ? current : candidate;
     }
     return null;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#issuecomment-5276481657

   Made some changes since the last review. Add some tests TestVariantShreddingAnalyzer tests: schema order-independence (admit and reject) and null-field handling (all-null not shredded, nulls+one-type still shreds).  Some branch cleanup in wider() and added some javadoc clarity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679299881


##########
parquet/src/test/java/org/apache/iceberg/parquet/TestVariantShreddingAnalyzer.java:
##########
@@ -471,6 +469,112 @@ public void testUuidFieldIsTrackedAndShredded() {
         .isInstanceOf(LogicalTypeAnnotation.UUIDLogicalTypeAnnotation.class);
   }
 
+  @Test
+  public void testMixedPrimitiveTypesFieldNotShredded() {
+    VariantMetadata meta = Variants.metadata("mixed", "keep");
+    ShreddedObject row1 = Variants.object(meta);
+    row1.put("mixed", Variants.of(42));
+    row1.put("keep", Variants.of(1));
+    ShreddedObject row2 = Variants.object(meta);
+    row2.put("mixed", Variants.of("text"));
+    row2.put("keep", Variants.of(2));
+
+    DirectAnalyzer analyzer = new DirectAnalyzer();
+    Type schema = analyzer.analyzeAndCreateSchema(List.of(row1, row2), 0);
+
+    assertThat(schema).isNotNull().isInstanceOf(GroupType.class);
+    GroupType typedValue = (GroupType) schema;
+    assertThat(typedValue.containsField("mixed")).isFalse();
+    assertThat(typedValue.containsField("keep")).isTrue();
+  }
+
+  @Test
+  public void testIntegerWideningAdmitsField() {

Review Comment:
   Where is the testDecimalWideningAdmitsField() :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3686379859


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +444,65 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+        admitted = merged;
       }
+      return admitted;
+    }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+    /**
+     * Merges {@code candidate} into the currently admitted type. Returns the wider type when both
+     * are in the same integer or decimal family, {@code candidate} when nothing is admitted yet,
+     * {@code current} when the types are identical, and null when the types are from incompatible
+     * families (including FLOAT vs DOUBLE, TIMESTAMPTZ vs TIMESTAMPTZ_NANOS, etc.).
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
-
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      if (current == candidate) {
+        return current;
       }
-
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
-    }
-
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+      PhysicalType widened = wider(current, candidate, INTEGER_TYPES);

Review Comment:
   I'll tell you I had this locally and chose not to do it because we had just two. Let me do this so it's better future looking.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3685770930


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric
+ *       widening; mixed-type fields remain in the residual {@code value}.
  *   <li>Integer types (INT8/16/32/64) and decimal types (DECIMAL4/8/16) are each promoted to the
- *       widest observed before competing with other types.
+ *       widest observed within their family.
  *   <li>Fields below {@code MIN_FIELD_FREQUENCY} are pruned. Above {@code MAX_SHREDDED_FIELDS}, the

Review Comment:
   FLOAT -> DOUBLE widening is lossless, but Iceberg's writer preserves the exact original type via the residual. Keeping them separate keeps that contract consistent, and matches TIMESTAMPTZ vs TIMESTAMPTZ_NANOS. I added a `testMixedFloatAndDoubleNotShredded` for it. Widening-on-write would be its own change, but not sure I'm seeing the benefit yet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679219293


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I generally like to avoid having multiple local variables that only work if they in certain combinations and I think we can make this a bit tighter if instead of tracking all these things separately we do something like.
   
   if admitted not set - set
   if admtted set and this type is wider, set admitted to wider type
   if admitted set and this is not  in the family - return null exit early on everything
   
   here is a quick draft i had the llm do
   
   ```java
   PhysicalType admittedType() {
     PhysicalType admitted = null;
     for (int i = 0; i < typeCounts.length; i++) {
       if (typeCounts[i] == 0) {
         continue;
       }
       PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
       if (merged == null) {
         // Mixed type families: do not shred this field.
         return null;
       }
       admitted = merged;
     }
     return admitted;
   }
   /**
    * Merges {@code candidate} into the currently admitted type.
    *
    * <p>Returns the wider type when both are in the same integer or decimal family, {@code
    * candidate} when nothing is admitted yet, {@code current} when the types are identical, and
    * null when the types are from incompatible families (including FLOAT vs DOUBLE).
    */
   private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
     if (current == null) {
       return candidate;
     }
     if (current == candidate) {
       return current;
     }
     if (isIntegerType(current) && isIntegerType(candidate)) {
       // INT8..INT64 are contiguous in PhysicalType declaration order.
       return current.ordinal() >= candidate.ordinal() ? current : candidate;
     }
     if (isDecimalType(current) && isDecimalType(candidate)) {
       // DECIMAL4..DECIMAL16 are contiguous in PhysicalType declaration order.
       return current.ordinal() >= candidate.ordinal() ? current : candidate;
     }
     return null;
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679260093


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I'm not sure on the ranking code there, originally I had it with enum ordinal but thats a spotless issue. Maybe we just have static ordered type lists for each family?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679219293


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   I generally like to avoid having multiple local variables that only work if they are set in certain combinations and I think we can make this a bit tighter if instead of tracking all these things separately we do something like.
   
   if admitted not set - set
   if admitted set and this is not in the family - return null exit early on everything
   if admtted set and this type is wider, set admitted to wider type
   
   here is a quick draft i had the llm do
   
   ```java
   PhysicalType admittedType() {
     PhysicalType admitted = null;
     for (int i = 0; i < typeCounts.length; i++) {
       if (typeCounts[i] == 0) {
         continue;
       }
       PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
       if (merged == null) {
         // Mixed type families: do not shred this field.
         return null;
       }
       admitted = merged;
     }
     return admitted;
   }
   /**
    * Merges {@code candidate} into the currently admitted type.
    *
    * <p>Returns the wider type when both are in the same integer or decimal family, {@code
    * candidate} when nothing is admitted yet, {@code current} when the types are identical, and
    * null when the types are from incompatible families (including FLOAT vs DOUBLE).
    */
   private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
     if (current == null) {
       return candidate;
     }
     if (current == candidate) {
       return current;
     }
     if (isIntegerType(current) && isIntegerType(candidate)) {
       return integerRank(current) >= integerRank(candidate) ? current : candidate;
     }
     if (isDecimalType(current) && isDecimalType(candidate)) {
       return decimalRank(current) >= decimalRank(candidate) ? current : candidate;
     }
     return null;
   }
   private static int integerRank(PhysicalType type) {
     return switch (type) {
       case INT8 -> 0;
       case INT16 -> 1;
       case INT32 -> 2;
       case INT64 -> 3;
       default -> throw new IllegalArgumentException("Not an integer type: " + type);
     };
   }
   private static int decimalRank(PhysicalType type) {
     return switch (type) {
       case DECIMAL4 -> 0;
       case DECIMAL8 -> 1;
       case DECIMAL16 -> 2;
       default -> throw new IllegalArgumentException("Not a decimal type: " + type);
     };
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3685782263


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -483,62 +459,56 @@ void observe(VariantValue value) {
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
+    PhysicalType admittedType() {
+      if (admittedTypeComputed) {
+        return admittedTypeCached;
       }
 
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
+      Set<PhysicalType> families = Sets.newHashSet();

Review Comment:
   Dropped widestInteger, widestDecimal, admittedTypeCached, and the families Set. Remaining state (typeCounts, decimal scale/digits, observation count) is used for schema build and pruning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#issuecomment-5126677729

   @Guosmilesmile thanks for taking a look. The raw column min/max you see (10000000000 for both) is because integer widening (INT8+INT64 → INT64 schema, preserved in admittedType() from the old getMostCommonType) combined with ShreddedVariantWriter.write doing an exact-type check means INT8/INT16/INT32 rows fall to residual value. But Iceberg's variant bounds handle this correctly: ParquetMetrics.value() invalidates the typed bounds whenever the value residual has any non-null entry, so filter pushdown doesn't use those raw stats. I recently fixed iceberg-go in (https://github.com/apache/iceberg-go/pull/1555) to match this behavior based on the spec. The spec permits either widening on write or falling to residual for out-of-schema types; Iceberg went with residual (see testMixedShredding in TestVariantWriters which relies on this to preserve exact width). This PR only adds the uniformity check above the widening logic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3679157700


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -51,10 +51,10 @@
  *
  * <ul>
  *   <li>Object fields are emitted in alphabetical order in the shredded schema.
- *   <li>Type selection picks the most common type with explicit tie-break priority (see {@link
- *       FieldInfo#TIE_BREAK_PRIORITY}), not enum ordinal.
+ *   <li>A field is admitted only if all observations fall into a single type family after numeric

Review Comment:
   May want to document that nested behavior and root behavior are slightly different here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3778617598


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +446,80 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+
+        admitted = merged;
+      }
+      return admitted;
+    }
+
+    /**
+     * Widens {@code current} with {@code candidate}, or null if they belong to different families.
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+      if (current == candidate) {
+        return current;
       }
 
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      List<PhysicalType> family = familyOf(current);
+      if (family == null) {
+        return null;
       }
 
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
+      return wider(current, candidate, family);
     }
 
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+    /** Returns the widening family for {@code type}, or null if none applies. */
+    private static List<PhysicalType> familyOf(PhysicalType type) {
+      if (INTEGER_TYPES.contains(type)) {
+        return INTEGER_TYPES;
+      }
+
+      if (DECIMAL_TYPES.contains(type)) {
+        return DECIMAL_TYPES;
+      }
+
+      return null;
     }
 
-    private static boolean isDecimalType(PhysicalType type) {
-      return type == PhysicalType.DECIMAL4
-          || type == PhysicalType.DECIMAL8
-          || type == PhysicalType.DECIMAL16;
+    /**
+     * Returns the wider of {@code first} and {@code second} within {@code family}, or null when
+     * {@code second} is not in the family. {@code first} is always a member.
+     */
+    private static PhysicalType wider(
+        PhysicalType first, PhysicalType second, List<PhysicalType> family) {
+      int firstIdx = family.indexOf(first);

Review Comment:
   paranoid nit :
   Precondition (firstIdx >= 0)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer merged PR #17424:
URL: https://github.com/apache/iceberg/pull/17424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Parquet, Spark, Flink: Type uniformity for variant shredding [iceberg]

Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17424:
URL: https://github.com/apache/iceberg/pull/17424#discussion_r3778888013


##########
parquet/src/main/java/org/apache/iceberg/parquet/VariantShreddingAnalyzer.java:
##########
@@ -475,83 +446,80 @@ void observe(VariantValue value) {
       typeCounts[type.ordinal()]++;
 
       // Track max precision and scale for decimal types
-      if (isDecimalType(type)) {
+      if (DECIMAL_TYPES.contains(type)) {
         if (value.asPrimitive().get() instanceof BigDecimal bd) {
           maxDecimalIntegerDigits = Math.max(maxDecimalIntegerDigits, bd.precision() - bd.scale());
           maxDecimalScale = Math.max(maxDecimalScale, bd.scale());
         }
       }
     }
 
-    PhysicalType getMostCommonType() {
-      if (mostCommonComputed) {
-        return mostCommonCached;
-      }
-
-      Map<PhysicalType, Integer> combinedCounts = Maps.newHashMap();
-
-      int integerTotalCount = 0;
-      PhysicalType mostCapableInteger = null;
-
-      int decimalTotalCount = 0;
-      PhysicalType mostCapableDecimal = null;
-
+    /**
+     * Returns the single type family that all observations fall into after numeric widening, or
+     * null if observations span multiple families.
+     */
+    PhysicalType admittedType() {
+      PhysicalType admitted = null;
       for (int i = 0; i < typeCounts.length; i++) {
-        int count = typeCounts[i];
-        if (count == 0) {
+        if (typeCounts[i] == 0) {
           continue;
         }
-        PhysicalType type = PHYSICAL_TYPES[i];
-
-        if (isIntegerType(type)) {
-          integerTotalCount += count;
-          if (mostCapableInteger == null
-              || INTEGER_PRIORITY.get(type) > INTEGER_PRIORITY.get(mostCapableInteger)) {
-            mostCapableInteger = type;
-          }
-        } else if (isDecimalType(type)) {
-          decimalTotalCount += count;
-          if (mostCapableDecimal == null
-              || DECIMAL_PRIORITY.get(type) > DECIMAL_PRIORITY.get(mostCapableDecimal)) {
-            mostCapableDecimal = type;
-          }
-        } else {
-          combinedCounts.put(type, count);
+        PhysicalType merged = mergeFamily(admitted, PHYSICAL_TYPES[i]);
+        if (merged == null) {
+          return null;
         }
+
+        admitted = merged;
+      }
+      return admitted;
+    }
+
+    /**
+     * Widens {@code current} with {@code candidate}, or null if they belong to different families.
+     */
+    private static PhysicalType mergeFamily(PhysicalType current, PhysicalType candidate) {
+      if (current == null) {
+        return candidate;
       }
 
-      if (mostCapableInteger != null) {
-        combinedCounts.put(mostCapableInteger, integerTotalCount);
+      if (current == candidate) {
+        return current;
       }
 
-      if (mostCapableDecimal != null) {
-        combinedCounts.put(mostCapableDecimal, decimalTotalCount);
+      List<PhysicalType> family = familyOf(current);
+      if (family == null) {
+        return null;
       }
 
-      // Pick the most common type with tie-breaking
-      mostCommonCached =
-          combinedCounts.entrySet().stream()
-              .max(
-                  Map.Entry.<PhysicalType, Integer>comparingByValue()
-                      .thenComparingInt(
-                          entry -> TIE_BREAK_PRIORITY.getOrDefault(entry.getKey(), -1)))
-              .map(Map.Entry::getKey)
-              .orElse(null);
-      mostCommonComputed = true;
-      return mostCommonCached;
+      return wider(current, candidate, family);
     }
 
-    private static boolean isIntegerType(PhysicalType type) {
-      return type == PhysicalType.INT8
-          || type == PhysicalType.INT16
-          || type == PhysicalType.INT32
-          || type == PhysicalType.INT64;
+    /** Returns the widening family for {@code type}, or null if none applies. */
+    private static List<PhysicalType> familyOf(PhysicalType type) {
+      if (INTEGER_TYPES.contains(type)) {
+        return INTEGER_TYPES;
+      }
+
+      if (DECIMAL_TYPES.contains(type)) {
+        return DECIMAL_TYPES;
+      }
+
+      return null;
     }
 
-    private static boolean isDecimalType(PhysicalType type) {
-      return type == PhysicalType.DECIMAL4
-          || type == PhysicalType.DECIMAL8
-          || type == PhysicalType.DECIMAL16;
+    /**
+     * Returns the wider of {@code first} and {@code second} within {@code family}, or null when
+     * {@code second} is not in the family. {@code first} is always a member.
+     */
+    private static PhysicalType wider(
+        PhysicalType first, PhysicalType second, List<PhysicalType> family) {
+      int firstIdx = family.indexOf(first);

Review Comment:
   Good paranoia



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org