You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "xndai (via GitHub)" <gi...@apache.org> on 2026/08/11 17:19:56 UTC

[PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

xndai opened a new pull request, #17609:
URL: https://github.com/apache/iceberg/pull/17609

   null_value_count was documented as only used for optional fields, but a required field nested within an optional struct is null whenever an ancestor value is null. Add definition of nullability by the full path from root, scope the required-field contract to a present enclosing struct, and require readers to rely on null_value_count == 0 rather than the required flag.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#discussion_r3770819625


##########
format/spec.md:
##########
@@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo
 | _optional_  | 2      | `upper_bound`             | Field type or `geo_upper` | all primitives or `variant`                   | Upper bound stored as the field's type, or `geo_upper` for geo types |
 | _optional_  | 3      | `tight_bounds`            | `boolean`                 | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values |
 | _optional_  | 4      | `value_count`             | `long`                    | all                                           | Number of values in the column (including null and NaN values) |
-| _optional_  | 5      | `null_value_count`        | `long`                    | optional fields                               | Number of null values in the column |
+| _optional_  | 5      | `null_value_count`        | `long`                    | nullable fields                               | Number of null values in the column |

Review Comment:
   This is a breaking change from the current behavior of our statistics.
   
   Right now, our null count matches what Parquet produces, which is a null count that is up to the number of values for the field. If a parent is null, it doesn't show up in the field-level count. It shows up in that the field has a different value count than the parent or the number of records in the data file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai commented on code in PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#discussion_r3770995142


##########
format/spec.md:
##########
@@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo
 | _optional_  | 2      | `upper_bound`             | Field type or `geo_upper` | all primitives or `variant`                   | Upper bound stored as the field's type, or `geo_upper` for geo types |
 | _optional_  | 3      | `tight_bounds`            | `boolean`                 | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values |
 | _optional_  | 4      | `value_count`             | `long`                    | all                                           | Number of values in the column (including null and NaN values) |
-| _optional_  | 5      | `null_value_count`        | `long`                    | optional fields                               | Number of null values in the column |
+| _optional_  | 5      | `null_value_count`        | `long`                    | nullable fields                               | Number of null values in the column |

Review Comment:
   > our null count matches what Parquet produces, which is a null count that is up to the number of values for the field. If a parent is null, it doesn't show up in the field-level count.
   
   I thought this was a bug, and even created a PR for that (https://github.com/apache/iceberg/pull/17560). We do see cases where planner was misleaded and produced wrong results. 
   
   I will take a look at #17413 and discuss there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5272826932

   Linking the discussion thread: https://lists.apache.org/thread/93c86nsm4k4zz3yv86mxrjnzw1blogb0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai closed pull request #17609: Spec: Clarify null_value_count applies to nullable fields
URL: https://github.com/apache/iceberg/pull/17609


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "Fokko (via GitHub) via issues" <is...@iceberg.apache.org>.
Fokko commented on code in PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#discussion_r4535484933


##########
format/spec.md:
##########
@@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo
 | _optional_  | 2      | `upper_bound`             | Field type or `geo_upper` | all primitives or `variant`                   | Upper bound stored as the field's type, or `geo_upper` for geo types |
 | _optional_  | 3      | `tight_bounds`            | `boolean`                 | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values |
 | _optional_  | 4      | `value_count`             | `long`                    | all                                           | Number of values in the column (including null and NaN values) |
-| _optional_  | 5      | `null_value_count`        | `long`                    | optional fields                               | Number of null values in the column |
+| _optional_  | 5      | `null_value_count`        | `long`                    | all                                           | Number of null values in the column |

Review Comment:
   I think this is different from what's discussed on the dev-list.
   
   For required top-level fields, there is no need to record null-count.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5273563219

   @xndai, @CTTY, let's try to keep discussion in one place. Right now, this is being discussed and addressed primarily in #17413. That's not a great place to finish the work because this is a separate change, but we can at least keep the discussion there until we identify the changes we need to fix this issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5943285817

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5318588701

   > @xndai in #17413 we concluded that we need to always track `null_value_count`. Would you like to re-open this PR and update the spec accordingly? I think the diff should look something like this:
   > 
   > ```
   > --- a/format/spec.md
   > +++ b/format/spec.md
   > @@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo
   >  | _optional_  | 2      | `upper_bound`             | Field type or `geo_upper` | all primitives or `variant`                   | Upper bound stored as the field's type, or `geo_upper` for geo types |
   >  | _optional_  | 3      | `tight_bounds`            | `boolean`                 | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values |
   >  | _optional_  | 4      | `value_count`             | `long`                    | all                                           | Number of values in the column (including null and NaN values) |
   > -| _optional_  | 5      | `null_value_count`        | `long`                    | optional fields                               | Number of null values in the column |
   > +| _optional_  | 5      | `null_value_count`        | `long`                    | all                                           | Number of null values in the column |
   >  | _optional_  | 6      | `nan_value_count`         | `long`                    | `float`, `double`                             | Number of NaN values in the column |
   >  | _optional_  | 7      | `avg_value_size_in_bytes` | `int`                     | `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size in memory (uncompressed) in bytes over non-null values to estimate memory consumption |
   > 
   > +`null_value_count` is included for `required` and `optional` fields because a field that is `required` in the table schema is **null** in rows where an `optional` parent struct is **null**.
   > +
   >  For example, stats for a `required` `int` field named `id` with field-id `2` are stored using:
   > 
   > 
   > @@ -834,8 +836,8 @@ For example, stats for a `required` `int` field named `id` with field-id `2` are
   >    10_402: optional int upper_bound; // type matches the field type (int)
   >    10_403: optional boolean tight_bounds;
   >    10_404: optional long value_count;
   > +  10_405: optional long null_value_count;
   > 
   > -  // null_value_count is only used for optional fields
   >    // nan_value_count is only used for float and double
   >    // avg_value_size_in_bytes is only used for variable length types
   >  }
   > @@ -902,6 +904,7 @@ For example, stats for a table with a required int, `id`, and an optional string
   >      10_402: optional int upper_bound;
   >      10_403: optional boolean tight_bounds;
   >      10_404: optional long value_count;
   > +    10_405: optional long null_value_count;
   >    }
   > ```
   
   Yes, I just re-open and push a new diff following your suggestion and remove the `nullable` term. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-5315974742

   @xndai in https://github.com/apache/iceberg/pull/17413 we concluded that we need to always track `null_value_count`. Would you like to re-open this PR and update the spec accordingly? I think the diff should look something like this:
   
   ```
   --- a/format/spec.md
   +++ b/format/spec.md
   @@ -822,10 +822,12 @@ Each stats struct holds statistics for one table field. It may contain the follo
    | _optional_  | 2      | `upper_bound`             | Field type or `geo_upper` | all primitives or `variant`                   | Upper bound stored as the field's type, or `geo_upper` for geo types |
    | _optional_  | 3      | `tight_bounds`            | `boolean`                 | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values |
    | _optional_  | 4      | `value_count`             | `long`                    | all                                           | Number of values in the column (including null and NaN values) |
   -| _optional_  | 5      | `null_value_count`        | `long`                    | optional fields                               | Number of null values in the column |
   +| _optional_  | 5      | `null_value_count`        | `long`                    | all                                           | Number of null values in the column |
    | _optional_  | 6      | `nan_value_count`         | `long`                    | `float`, `double`                             | Number of NaN values in the column |
    | _optional_  | 7      | `avg_value_size_in_bytes` | `int`                     | `string`, `binary`, `variant`, `geometry`, `geography` | Avg value size in memory (uncompressed) in bytes over non-null values to estimate memory consumption |
   
   +`null_value_count` is included for `required` and `optional` fields because a field that is `required` in the table schema is **null** in rows where an `optional` parent struct is **null**.
   +
    For example, stats for a `required` `int` field named `id` with field-id `2` are stored using:
   
   
   @@ -834,8 +836,8 @@ For example, stats for a `required` `int` field named `id` with field-id `2` are
      10_402: optional int upper_bound; // type matches the field type (int)
      10_403: optional boolean tight_bounds;
      10_404: optional long value_count;
   +  10_405: optional long null_value_count;
   
   -  // null_value_count is only used for optional fields
      // nan_value_count is only used for float and double
      // avg_value_size_in_bytes is only used for variable length types
    }
   @@ -902,6 +904,7 @@ For example, stats for a table with a required int, `id`, and an optional string
        10_402: optional int upper_bound;
        10_403: optional boolean tight_bounds;
        10_404: optional long value_count;
   +    10_405: optional long null_value_count;
      }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spec: Clarify null_value_count applies to nullable fields [iceberg]

Posted by "xndai (via GitHub)" <gi...@apache.org>.
xndai commented on PR #17609:
URL: https://github.com/apache/iceberg/pull/17609#issuecomment-6042628210

   @rdblue can you please review it again otherwise PR is going to be closed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org