You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hemanthboyina (via GitHub)" <gi...@apache.org> on 2026/04/15 13:36:37 UTC

[PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

hemanthboyina opened a new pull request, #15979:
URL: https://github.com/apache/iceberg/pull/15979

   The total_record_count field in partition statistics was always left as NULL
   
   However, as discussed in #12098, with V3 deletion vectors (DVs), the total can be computed cheaply from manifest metadata without scanning any data files. DV cardinalities are exact, each bitmap position maps to one deleted row. so data_record_count - position_delete_record_count yields a precise total
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4617998921

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4312963038

   Calling my answer to a community feedback is a bit thin :) Let's wait a bit if someone else chimes in.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #15979: Core: Populate total_record_count in partition stats when computable from metadata
URL: https://github.com/apache/iceberg/pull/15979


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4311865434

   @gaborkaszab I have raised this on dev@ as you suggested. The community feedback supports computing total_record_count in core when unambiguous, leaving null otherwise. I've also pushed a simplified version that removes the object copying by allowing null for optional fields in BasePartitionStatistics.internalSet. Would appreciate another look.
   
   cc @pvary 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3106981398


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {
+      boolean hasEqualityDeletes =
+          partitionStats.equalityDeleteRecordCount() != null
+              && partitionStats.equalityDeleteRecordCount() > 0;
+      boolean hasV2PositionDeleteFiles =
+          partitionStats.positionDeleteFileCount() != null
+              && partitionStats.positionDeleteFileCount() > 0;
+
+      // Create a fresh BasePartitionStatistics to avoid projection issues on read-back objects

Review Comment:
   BasePartitionStatistics#internalSet has a null check 
   if (value == null) 
   { return; }
   this skips null values. I think this exists to prevent NPE from unboxing required premitive fields. But it means we cannot reset total_record_count back to null on objects that already have a non-null value from a previous stats computation. Fresh BasePartitionStatistics objects start with totalRecordCount = null by default, so we copy all fields and only set total_record_count when it's computable.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4367997699

   Replied on the mailing list
   
   ```
   I personally think these changes are not required for the following reasons:
   
   Consistency: We can apply this same logic to positional deletes, not just DVs (Peter also mentioned this).
   
   Spec Adherence: The total_record_count should reflect an accurate value in all cases. The spec defines it as the "Accurate count of records in a partition after applying deletes if any," which implies engines should account for equality deletes as well.
   
   Redundancy: The current PR simply calculates total_record = data record - dv record. This is easily derived by the user from other existing stats, so specific handling isn't necessary.
   
   Existing Logic: We don't force total_record = data record when no delete files exist. Since those stats are already easily inferred by the user, we should maintain that same approach here.
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4371995087

   Thank you for sharing your view, @ajantha-bhat !
   
   Let me sum up how I understand the situation so far:
   - Calculating total_record_count as a derived value on the write path probably does't add much value, because that a) can't work on all use-cases (eq-deletes, v2 pos-deletes) and b) could easily be calculated by the user engines if the partition doesn't have or has only v3 DVs.
   - Calculating total_record_count for all use cases requires a full data scan applying the deletes. Engines might be able to do this when performing a full scan for other purposes, but this is very speculative, I don't think any engine would do this.
   
   I'd lean towards dropping it from the spec then, because I don't see any motivation to write it TBH. Keeping it there so that some engine in the future might write it doesn't make sense to me. WDYT @ajantha-bhat, @pvary  ?
   
   Additionally, we could consider if it's still valuable to calculate it as a derived value on the read path, though.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4686365885

   This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4282289244

   Hi @hemanthboyina ,
   I gave this PR some thought the recent days, and frankly I'm a bit hesitant here. The reason is that this stat we are populating now is a derived stat, so all the information is there in the existing stats to produce this new one. Additionally, I think the code needed for this isn't that simple either. I'm wondering how much sense it makes to use the Iceberg lib to produce this stat or rather we simply say that we leave it to the engines to calculate it on their own. I know this is part of the spec now, but maybe we'd like to revisit that.
   To get better coverage, would you mind raising this to the dev@ list? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4282457907

   thanks @gaborkaszab , regarding the code complexity, with the internalSet null check fix which I mentioned, the implementation simplifies to a straightforward in-place loop (no object copying), which I think addresses the code complexity concern.
   
   I can push that simplified version if you'd like to take another look before escalating to dev@. Otherwise happy to raise the discussion there.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4342973474

   I gave this some further thoughts. I think, since this is a derived stat, we shouldn't write this and also we shouldn't have this in the spec either as the content of the stat file. Leaving the door open for engines performing a full scan to write the highest precision total_record_count stat seems very speculative.
   I'd rather populate this on the read path if all the conditions are met instead of writing it to the stat file.
   
   I see @pvary is on another opinion. We can wait if there is anyone else chiming in.
   In the meantime let me ping @ajantha-bhat one more time to see if he can share some additional context on the introduction of this stat.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3106979020


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {

Review Comment:
   I think for v3 tables using DVs, all partitions will have total_record_count populated since there are no equality deletes or V2 position delete files. The partial population occurs mainly for V2→V3 upgraded tables that still have legacy delete files.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#issuecomment-4276353119

   @gaborkaszab While investigating why the copy was needed, I found that BasePartitionStatistics#internalSet ignores null values to all fields, including the optional ones (total_record_count, last_updated_at, last_updated_snapshot_id) where null is a valid value per the spec. Can I raise a different PR to fix this in internalSet(), and then rebase this PR to use the simpler in-place approach without copying.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3106962161


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {
+      boolean hasEqualityDeletes =
+          partitionStats.equalityDeleteRecordCount() != null
+              && partitionStats.equalityDeleteRecordCount() > 0;
+      boolean hasV2PositionDeleteFiles =
+          partitionStats.positionDeleteFileCount() != null
+              && partitionStats.positionDeleteFileCount() > 0;
+

Review Comment:
   The copy is needed for all partitions, not just the computable ones. Because BasePartitionStatistics#internalSet has a null check that ignores null values. this means we can't reset a stale total_record_count on an existing object that already carries non-null value from previous stats file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3110768363


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {

Review Comment:
   V3 tables could still have eq-deletes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3111581026


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {

Review Comment:
   Right, a V3 table can use both DVs and equality deletes. To clarify, total_record_count is populated per-partition only when that partition has no equality deletes and no V2 position delete files.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "hemanthboyina (via GitHub)" <gi...@apache.org>.
hemanthboyina commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3106964885


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -608,7 +682,9 @@ private static void liveEntry(PartitionStatistics stats, ContentFile<?> file, Sn
       updateSnapshotInfo(stats, snapshot.snapshotId(), snapshot.timestampMillis());
     }
 
-    // Note: Not computing the `TOTAL_RECORD_COUNT` for now as it needs scanning the data.
+    // Note: total_record_count is populated by populateTotalRecordCount() after all

Review Comment:
   the decision of whether total_record_count can be computed depends on final aggregate state of a partition. Specifically on equalityDeleteRecordCount and positionDeleteFileCount. these values are accumulated entry by entry in liveEntry(). So the total can only be reliably determined after all entries are processed and merged.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Populate total_record_count in partition stats when computable from metadata [iceberg]

Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #15979:
URL: https://github.com/apache/iceberg/pull/15979#discussion_r3093387568


##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {
+      boolean hasEqualityDeletes =
+          partitionStats.equalityDeleteRecordCount() != null
+              && partitionStats.equalityDeleteRecordCount() > 0;
+      boolean hasV2PositionDeleteFiles =
+          partitionStats.positionDeleteFileCount() != null
+              && partitionStats.positionDeleteFileCount() > 0;
+

Review Comment:
   Why not skip the rest of the loop here if there are eq or v2 pos deletes?



##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -608,7 +682,9 @@ private static void liveEntry(PartitionStatistics stats, ContentFile<?> file, Sn
       updateSnapshotInfo(stats, snapshot.snapshotId(), snapshot.timestampMillis());
     }
 
-    // Note: Not computing the `TOTAL_RECORD_COUNT` for now as it needs scanning the data.
+    // Note: total_record_count is populated by populateTotalRecordCount() after all

Review Comment:
   Can there be a way to calculate this stat as part of `computeStats` and/or `computeAndMergeStatsIncremental`?
   Anyway these comments aren't that useful here, at least for me.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {

Review Comment:
   I wonder how useful such a stat can be that is populated for some partitions but might not be populated for others.



##########
core/src/main/java/org/apache/iceberg/PartitionStatsHandler.java:
##########
@@ -559,6 +564,75 @@ private static List<PartitionStatistics> sortStatsByPartition(
     return entries;
   }
 
+  /**
+   * Populates {@code total_record_count} for partitions where it can be reliably computed from
+   * metadata without scanning data files.
+   *
+   * <p>Per the spec, {@code total_record_count} should stay optional (NULL means unknown). It can
+   * be computed when a partition has no equality deletes and no V2 position delete files — i.e.,
+   * deletes are tracked exclusively via deletion vectors (DVs) or the partition has no deletes at
+   * all. In that case, the total is: {@code data_record_count - position_delete_record_count}.
+   *
+   * <p>DV cardinalities are exact (each position maps to one deleted row), so the subtraction
+   * yields a precise total without reading any data files.
+   *
+   * @param stats the computed partition statistics to enrich
+   */
+  @VisibleForTesting
+  static List<PartitionStatistics> populateTotalRecordCount(Collection<PartitionStatistics> stats) {
+    List<PartitionStatistics> result = Lists.newArrayListWithExpectedSize(stats.size());
+    for (PartitionStatistics partitionStats : stats) {
+      boolean hasEqualityDeletes =
+          partitionStats.equalityDeleteRecordCount() != null
+              && partitionStats.equalityDeleteRecordCount() > 0;
+      boolean hasV2PositionDeleteFiles =
+          partitionStats.positionDeleteFileCount() != null
+              && partitionStats.positionDeleteFileCount() > 0;
+
+      // Create a fresh BasePartitionStatistics to avoid projection issues on read-back objects

Review Comment:
   I'm not sure I get this. Could you explain more why the copy is needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org