You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "GayathriSrividya (via GitHub)" <gi...@apache.org> on 2026/06/18 08:23:21 UTC
[PR] fix: add iceberg_type column and filter all SqlCatalog table operations [iceberg-python]
GayathriSrividya opened a new pull request, #3525:
URL: https://github.com/apache/iceberg-python/pull/3525
closes #3337
## Summary
Adds the `iceberg_type` column to the `IcebergTables` ORM model (also superseding #3263) and extends the type filter to **all** table operations, matching Java's `JdbcCatalog` behavior.
## Root cause
PR #3263 proposed adding `iceberg_type` to `list_tables`, but was not yet merged. Meanwhile #3337 identified that even after that fix, `load_table`, `drop_table`, `rename_table`, and `commit_table` would still lack the filter. View rows written by iceberg-java or iceberg-rust can therefore bleed into Python's table operations.
Java's JdbcCatalog applies `WHERE (iceberg_type = 'TABLE' OR iceberg_type IS NULL)` to every table lookup ([JdbcUtil.java#L168](https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java#L168)).
## Changes
### pyiceberg/catalog/sql.py
- Add `iceberg_type: Mapped[str | None]` column to `IcebergTables` model.
- Add `ICEBERG_TABLE_TYPE = "TABLE"` constant.
- Add idempotent `ALTER TABLE iceberg_tables ADD COLUMN iceberg_type` migration in `_ensure_tables_exist` (backward-compatible; catches `OperationalError`/`ProgrammingError` if column already exists).
- Set `iceberg_type=ICEBERG_TABLE_TYPE` on all `IcebergTables` inserts: `create_table`, `register_table`, `commit_table` (new table path).
- Apply `WHERE (iceberg_type = 'TABLE' OR iceberg_type IS NULL)` filter to: `load_table`, `drop_table`, `rename_table`, `commit_table`, `list_tables`.
### tests/catalog/test_sql.py
- Add `TestIcebergTypeFilter` class with 5 regression tests:
- `test_iceberg_type_set_on_create` — verifies new tables get `iceberg_type='TABLE'`
- `test_list_tables_excludes_view_rows`
- `test_load_table_ignores_view_rows`
- `test_drop_table_ignores_view_rows`
- `test_rename_table_ignores_view_rows`
## Validation
- `make lint` ✓
- `pytest tests/catalog/test_sql.py` → 17 passed ✓
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: add iceberg_type column and filter all SqlCatalog table operations [iceberg-python]
Posted by "ebyhr (via GitHub)" <gi...@apache.org>.
ebyhr commented on code in PR #3525:
URL: https://github.com/apache/iceberg-python/pull/3525#discussion_r3441152988
##########
pyiceberg/catalog/sql.py:
##########
@@ -147,6 +153,17 @@ def _ensure_tables_exist(self) -> None:
self.create_tables()
return
+ # Idempotently add iceberg_type column if it does not exist yet
+ # (backward-compatible migration for databases created before this column was introduced).
+ # Older SQLite versions do not support "ADD COLUMN IF NOT EXISTS", so we catch and ignore
+ # the error if the column is already present.
+ with Session(self.engine) as session:
+ try:
+ session.execute(text("ALTER TABLE iceberg_tables ADD COLUMN iceberg_type VARCHAR(255)"))
Review Comment:
I don't think we should add a new column in `_ensure_tables_exist` method. It is counterintuitive. The initialization logic in #3263 looks better than this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: add iceberg_type column and filter all SqlCatalog table operations [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya commented on PR #3525:
URL: https://github.com/apache/iceberg-python/pull/3525#issuecomment-4966228836
> btw there was already another PR #3263
Closing this PR as superseded by PR #3263 direction and reviewer feedback. I’ll follow up with a focused change that keeps the #3263 initialization/migration approach and only adds the missing iceberg_type filtering across SqlCatalog table operations from issue #3337.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: add iceberg_type column and filter all SqlCatalog table operations [iceberg-python]
Posted by "GayathriSrividya (via GitHub)" <gi...@apache.org>.
GayathriSrividya closed pull request #3525: fix: add iceberg_type column and filter all SqlCatalog table operations
URL: https://github.com/apache/iceberg-python/pull/3525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] fix: add iceberg_type column and filter all SqlCatalog table operations [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #3525:
URL: https://github.com/apache/iceberg-python/pull/3525#issuecomment-4763926837
btw there was already another PR #3263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org