You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "huan233usc (via GitHub)" <gi...@apache.org> on 2026/08/28 05:11:31 UTC
[PR] Core, OpenAPI: Preserve avg value sizes across rewrites [iceberg]
huan233usc opened a new pull request, #17855:
URL: https://github.com/apache/iceberg/pull/17855
## Summary
Follow-up to #17451. Average value sizes were lost when content files crossed the planned REST scan JSON boundary, while `rewrite_table_path` retained a stale average for the position-delete `file_path` column after rewriting that path.
- Preserve `avg-value-sizes` for both data and delete files in `ContentFileParser` JSON round trips and document the optional field in the REST OpenAPI schema.
- Remove the `file_path` average value size when rewriting position-delete path bounds, preserve unrelated averages, and normalize an empty result to null.
## Test plan
- [x] `./gradlew :iceberg-core:test --tests org.apache.iceberg.TestContentFileParser --tests org.apache.iceberg.TestFileScanTaskParser --tests org.apache.iceberg.TestDataTaskParser --tests org.apache.iceberg.TestRewriteTablePathUtil`
- [x] `./gradlew :iceberg-core:spotlessCheck`
- [x] Verified generated Python syntax and OpenAPI YAML syntax.
The core tests used RoaringBitmap 1.6.19 temporarily because the configured Maven mirror returned 403 for 1.6.20; the dependency version is unchanged in this PR.
---
**AI Disclosure**
- Model: [unknown - human to fill in]
- Platform/Tool: Cursor
- Human Oversight: unreviewed
- Prompt Summary: Implement and open a PR for the two follow-ups from #17451: REST content-file JSON round trips and rewrite-table-path average-size handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#issuecomment-5448981991
Hi @singhpk234 for quick review -- I just realize that this PR was not marked as draft when I am double checking the generated code vs spec. Will asked for another review when I finish my self validation. Sry for confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#issuecomment-5448935666
Split the rewrite_table_path follow-up into #17856. This PR is now REST JSON / OpenAPI only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho merged PR #17855:
URL: https://github.com/apache/iceberg/pull/17855
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#issuecomment-5498658090
Updated the PR, PTAL @singhpk234 @nastra @szehon-ho Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#discussion_r3919882601
##########
open-api/rest-catalog-open-api.py:
##########
@@ -1101,6 +1109,11 @@ class ContentFile(BaseModel):
None, alias='split-offsets', description='List of splittable offsets'
)
sort_order_id: int | None = Field(None, alias='sort-order-id')
+ content_stats: dict[str, FieldStatistics] | None = Field(
+ None,
+ alias='content-stats',
+ description='Container struct for per-field metrics structs',
Review Comment:
Could we document that `content-stats` keys are decimal string field IDs, ideally with an example? The parser requires numeric keys, while the OpenAPI schema currently allows arbitrary strings. Fine to address in a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes across rewrites [iceberg]
Posted by "singhpk234 (via GitHub)" <gi...@apache.org>.
singhpk234 commented on code in PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#discussion_r3878119988
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -5123,6 +5123,25 @@ components:
"values": [ 100, 200 ]
}
+ IntegerMap:
+ type: object
+ properties:
+ keys:
+ type: array
+ items:
+ $ref: '#/components/schemas/IntegerTypeValue'
+ description: "List of integer column ids for each corresponding value"
+ values:
+ type: array
+ items:
+ $ref: '#/components/schemas/IntegerTypeValue'
+ description: "List of integer values, matched to 'keys' by index"
+ example:
+ {
+ "keys": [ 1, 2 ],
+ "values": [ 8, 16 ]
+ }
Review Comment:
can't we use ValueMap ? like what we use for other stats
https://github.com/apache/iceberg/blob/ef8a69dc3dab5858d612770062bbf5a6043170f0/open-api/rest-catalog-open-api.yaml#L5126
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -5222,6 +5241,12 @@ components:
description: "List of splittable offsets"
sort-order-id:
type: integer
+ avg-value-sizes:
+ allOf:
+ - $ref: '#/components/schemas/IntegerMap'
+ description:
+ Map of column id to average value size in memory (uncompressed) in
+ bytes over non-null values
Review Comment:
would this be part of data file ? like lower / upper bound are ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "huan233usc (via GitHub)" <gi...@apache.org>.
huan233usc commented on code in PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#discussion_r3907211169
##########
open-api/rest-catalog-open-api.yaml:
##########
@@ -5123,6 +5123,25 @@ components:
"values": [ 100, 200 ]
}
+ IntegerMap:
+ type: object
+ properties:
+ keys:
+ type: array
+ items:
+ $ref: '#/components/schemas/IntegerTypeValue'
+ description: "List of integer column ids for each corresponding value"
+ values:
+ type: array
+ items:
+ $ref: '#/components/schemas/IntegerTypeValue'
+ description: "List of integer values, matched to 'keys' by index"
+ example:
+ {
+ "keys": [ 1, 2 ],
+ "values": [ 8, 16 ]
+ }
Review Comment:
updated the OpenAPI to match v4 field statistics (content-stats + avg-value-size-in-bytes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Preserve avg value sizes in REST JSON [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on PR #17855:
URL: https://github.com/apache/iceberg/pull/17855#issuecomment-5535320946
Merged, thanks @huan233usc and @singhpk234 for additional early review !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org