You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "dbnl-tjs (via GitHub)" <gi...@apache.org> on 2026/02/12 21:25:27 UTC
[I] PyIceberg projected scan fails with "Could not find field with id" on partitioned table [iceberg-python]
dbnl-tjs opened a new issue, #3040:
URL: https://github.com/apache/iceberg-python/issues/3040
### Apache Iceberg version
0.10.0 (latest release)
### Please describe the bug 🐞
Projected reads on a partitioned table can fail with:
`ValueError: Could not find field with id: 2`
This occurs when scanning with `selected_fields` that exclude the partition source column (for example projecting only `field1` while the table is partitioned by `day(timestamp)`).
## Expected behavior
A projected read should succeed even if the projection does not include partition source columns.
## Actual behavior
`table.scan(..., selected_fields=(...)).to_arrow()` fails in PyIceberg planning/execution with:
```
ValueError: Could not find field with id: 2
```
## Environment
- Python: 3.14.2
- pyiceberg: 0.10.0
- pyiceberg-core: 0.6.0
- backend: SQL catalog
- IO impl: `pyiceberg.io.pyarrow.PyArrowFileIO`
## Repro
```python
from datetime import UTC, datetime
from pathlib import Path
import tempfile
import pyarrow as pa
from pyiceberg.catalog import load_catalog
from pyiceberg.expressions import AlwaysTrue
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import DayTransform
from pyiceberg.types import NestedField, StringType, TimestamptzType
with tempfile.TemporaryDirectory(prefix="pyiceberg-repro-") as tmp:
tmp_path = Path(tmp)
db_path = tmp_path / "catalog.db"
warehouse_path = tmp_path / "warehouse"
warehouse_path.mkdir(parents=True, exist_ok=True)
catalog = load_catalog(
"default",
**{
"type": "sql",
"uri": f"sqlite:///{db_path}",
"warehouse": warehouse_path.as_uri(),
"py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
},
)
catalog.create_namespace_if_not_exists("ns")
schema_v1 = Schema(
NestedField(field_id=1, name="timestamp", field_type=TimestamptzType(), required=True),
NestedField(field_id=2, name="value", field_type=StringType(), required=False),
)
spec = PartitionSpec(
PartitionField(source_id=1, field_id=1000, transform=DayTransform(), name="_day"),
)
table = catalog.create_table("ns.tbl", schema=schema_v1, partition_spec=spec)
# File 1: old schema (no new_col)
table.append(
pa.Table.from_pylist(
[{"timestamp": datetime(2025, 1, 1, tzinfo=UTC), "value": "old"}],
schema=pa.schema(
[
pa.field("timestamp", pa.timestamp("us", tz="UTC"), nullable=False),
pa.field("value", pa.string()),
]
),
)
)
# Evolve schema
with table.update_schema() as u:
u.add_column("new_col", StringType())
table = catalog.load_table("ns.tbl")
# File 2: new schema (has new_col)
table.append(
pa.Table.from_pylist(
[{"timestamp": datetime(2025, 1, 2, tzinfo=UTC), "value": "new", "new_col": "x"}],
schema=pa.schema(
[
pa.field("timestamp", pa.timestamp("us", tz="UTC"), nullable=False),
pa.field("value", pa.string()),
pa.field("new_col", pa.string()),
]
),
)
)
# Repro on affected versions:
print(table.scan(row_filter=AlwaysTrue(), selected_fields=("new_col",)).to_arrow())
# Workaround:
# print(table.scan(row_filter=AlwaysTrue(), selected_fields=("new_col", "timestamp")).to_arrow())
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg projected scan fails with "Could not find field with id" on partitioned table [iceberg-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on issue #3040:
URL: https://github.com/apache/iceberg-python/issues/3040#issuecomment-3941890861
Thanks for the repro script! I think this is fixed in the new 0.11.0 release. Could you give it a try?
i tested locally and it works (`uv run --with pyiceberg==0.11.0 repro.py`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg projected scan fails with "Could not find field with id" on partitioned table [iceberg-python]
Posted by "dbnl-tjs (via GitHub)" <gi...@apache.org>.
dbnl-tjs closed issue #3040: PyIceberg projected scan fails with "Could not find field with id" on partitioned table
URL: https://github.com/apache/iceberg-python/issues/3040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] PyIceberg projected scan fails with "Could not find field with id" on partitioned table [iceberg-python]
Posted by "dbnl-tjs (via GitHub)" <gi...@apache.org>.
dbnl-tjs commented on issue #3040:
URL: https://github.com/apache/iceberg-python/issues/3040#issuecomment-3993161667
Confirmed fixed, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org