You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kumarpritam863 (via GitHub)" <gi...@apache.org> on 2025/12/12 03:04:28 UTC
[PR] Avoid always adding columns as optional on schema evolution [iceberg]
kumarpritam863 opened a new pull request, #14827:
URL: https://github.com/apache/iceberg/pull/14827
### Problem
When performing schema evolution (adding a new field), the code currently calls `addColumn()` → `addInternalColumn()`, which **always** marks the new column as `OPTIONAL` (i.e. nullable = true), regardless of what the Connect schema says.
This behaviour is **inconsistent** with table creation:
- On **initial table creation** we correctly respect `schema.isOptional()` (and the related configuration) to decide whether a column should be `REQUIRED` or `OPTIONAL`.
- On **schema evolution** we ignore that and force the new column to be `OPTIONAL`.
As a result:
- If you create the table directly with schema **v2** (which already contains the new field), the column is created with the correct nullability.
- If you create the table with **v1** and later evolve to **v2**, the same column is added as `OPTIONAL` → different physical schema.
### Solution
This PR aligns schema-evolution nullability handling with the logic used during initial table creation.
Now the same code path (respecting `ConnectSchema.isOptional()` and the configured default for required fields) is used in **both** cases, eliminating the dual behaviour and guaranteeing that the resulting Parquet files have identical column nullability no matter whether the field was present at creation time or added later via schema evolution.
### Result
- Consistent Parquet schema across creation and evolution paths
- New columns respect the nullability declared in the Connect schema
- No more surprise `OPTIONAL` columns when the source schema says the field is required
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Avoid always adding columns as optional on schema evolution [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/iceberg/pull/14827#issuecomment-3736515772
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Avoid always adding columns as optional on schema evolution [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14827: Avoid always adding columns as optional on schema evolution
URL: https://github.com/apache/iceberg/pull/14827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Avoid always adding columns as optional on schema evolution [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14827:
URL: https://github.com/apache/iceberg/pull/14827#issuecomment-3770516901
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org