You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "VABVAT (via GitHub)" <gi...@apache.org> on 2026/10/10 06:48:42 UTC

[PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

VABVAT opened a new pull request, #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390

   ## Which issue does this PR close?
   
   - Closes #2749
   
   ## What changes are included in this PR?
   
   This PR updates Glue schema conversion so AWS Glue `StorageDescriptor.columns` is populated only from the current Iceberg schema.
   
   Previously, `GlueSchemaBuilder::from_iceberg` converted the current schema and then also converted historical schemas from table metadata. Those historical fields were written to Glue with `iceberg.field.current=false`, but Glue does not use that parameter to filter visible columns, which could cause duplicate columns to appear in the Glue Console after schema evolution.
   
   Historical schemas are still preserved in Iceberg table metadata. This change only prevents them from being projected into Glue's visible column list.
   
   ## Are these changes tested?
   
   Yes.
   
   Added a regression test covering schema evolution with historical schemas, and verified the Glue schema output contains only current schema fields.
   
   Ran locally:
   
   ```shell
   cargo test -p iceberg-catalog-glue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

Posted by "VABVAT (via GitHub) via issues" <is...@iceberg.apache.org>.
VABVAT commented on PR #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390#issuecomment-6806096474

   > Thanks, this matches the Java converter. One difference left: `schemas_iter()` iterates a `HashMap`, so when two older schemas carry a dropped column with the same name but a different id or type, the one that ends up in Glue can change from one commit to the next. Java walks `metadata.schemas()` in order, so the oldest schema wins. Sorting the older schemas by id before visiting them makes it deterministic:
   > 
   > ```rust
   > let mut schemas: Vec<_> = metadata
   >     .schemas_iter()
   >     .filter(|schema| schema.schema_id() != current_schema.schema_id())
   >     .collect();
   > schemas.sort_by_key(|schema| schema.schema_id());
   > ```
   > 
   > A test case for it: `legacy` as `int` in schema 0, promoted to `long` in schema 1, dropped in the current schema. Glue should get `int`.
   
   Hey @David-Banquet, Sorry for keeping this PR open, totally forgot about it.
   Historical schemas are now sorted by schema ID before visiting them, making it deterministic.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

Posted by "David-Banquet (via GitHub) via issues" <is...@iceberg.apache.org>.
David-Banquet commented on PR #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390#issuecomment-6805851289

   Thanks, this matches the Java converter. One difference left: `schemas_iter()` iterates a `HashMap`, so when two older schemas carry a dropped column with the same name but a different id or type, the one that ends up in Glue can change from one commit to the next. Java walks `metadata.schemas()` in order, so the oldest schema wins. Sorting the older schemas by id before visiting them makes it deterministic:
   
   ```rust
   let mut schemas: Vec<_> = metadata
       .schemas_iter()
       .filter(|schema| schema.schema_id() != current_schema.schema_id())
       .collect();
   schemas.sort_by_key(|schema| schema.schema_id());
   ```
   
   A test case for it: `legacy` as `int` in schema 0, promoted to `long` in schema 1, dropped in the current schema. Glue should get `int`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

Posted by "David-Banquet (via GitHub)" <gi...@apache.org>.
David-Banquet commented on PR #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390#issuecomment-6109549770

   Thanks for picking this up again.
   
   This drops the older schemas entirely, so columns removed from the table also disappear from Glue. The Java catalog keeps them: `toColumns` in `IcebergToGlueConverter` writes the current schema first, then the older schemas with `iceberg.field.current=false`, and skips any name it has already added. The duplicates come from that missing name check, so keeping the loop and adding a `HashSet` of top-level names already written would fix #2749 and keep the Glue output the same as Java's. A table written by both Spark and iceberg-rust would otherwise see its Glue columns change depending on which engine committed last.
   
   If the maintainers prefer current-schema-only, that's a fair choice too, but it's a behavior change from Java and probably worth saying so in the PR description.
   
   Two smaller things:
   - The new test only adds a column, where both approaches give the same result. A case that drops a column (and one where a field keeps its name across schemas with a different type) would show which behavior is intended.
   - `builder.is_current = false;` and its comment can go: the fields are private and nothing reads them after `from_iceberg` returns.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@iceberg.apache.org>.
github-actions[bot] commented on PR #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390#issuecomment-6782985842

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] fix(glue): Exclude historical schemas from Glue columns [iceberg-rust]

Posted by "David-Banquet (via GitHub) via issues" <is...@iceberg.apache.org>.
David-Banquet commented on PR #3390:
URL: https://github.com/apache/iceberg-rust/pull/3390#issuecomment-6804623494

   @VABVAT are you still working on this? The open point is the one from my comment above: keep the older schemas, but write the current schema first, then the older ones with `iceberg.field.current=false`, skipping any name already added, as `IcebergToGlueConverter.toColumns` does in Java. That fixes the duplicates without dropping removed columns from Glue.
   
   If you don't have time for it, I can open a PR with that approach and keep you as co-author.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org