You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "wjixiang (via GitHub)" <gi...@apache.org> on 2026/07/17 06:07:10 UTC

[I] Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found" [iceberg-rust]

wjixiang opened a new issue, #2837:
URL: https://github.com/apache/iceberg-rust/issues/2837

   ### Apache Iceberg Rust version
   
   None
   
   ### Describe the bug
   
   ## Summary
   
   A table with a user data column named `pos` writes fine and appears correctly in the catalog schema, but **any scan that projects it fails** with `External(Unexpected =>
    "field not found")`. Renaming the column to anything else (e.g. `position`) makes it readable again. The same applies to `file_path`.
   
   The bare names `pos` and `file_path` (no leading underscore) are reserved as metadata-column names (`RESERVED_COL_NAME_DELETE_FILE_POS`, `RESERVED_COL_NAME_DELETE_FILE_P
   ATH`), and the data-table scan resolves column names with **metadata-first precedence and no fallback to a same-named data column**, so a real data column named `pos`/`f
   ile_path` is shadowed.
   
   ## Minimal reproduction (pure `iceberg` crate, no catalog/storage)
   
   Uses only the public `metadata_columns` API — runs in milliseconds with no infrastructure:
   
   ```rust
   use iceberg::metadata_columns::{
       get_metadata_field_id, is_metadata_column_name,
       RESERVED_COL_NAME_DELETE_FILE_POS, RESERVED_FIELD_ID_DELETE_FILE_POS,
   };
   
   fn main() {
       // The reserved name is the bare `pos` (not the spec's `_pos`).
       assert_eq!(RESERVED_COL_NAME_DELETE_FILE_POS, "pos");
   
       // A user data column named `pos` is misclassified as a metadata column.
       // Per the Iceberg spec only `_`-prefixed names are metadata columns,
       // so this should be `false`.
       assert!(is_metadata_column_name("pos"));
       assert!(is_metadata_column_name("file_path"));
   
       // So a data column `pos` resolves to the reserved delete-file field id
       // (i32::MAX - 102) instead of its real schema field id.
       assert_eq!(
           get_metadata_field_id("pos").unwrap(),
           RESERVED_FIELD_ID_DELETE_FILE_POS,
       );
   }
   ```
   
   ## Expected behavior
   
   A user data column named `pos` should be readable. The [Iceberg spec](https://iceberg.apache.org/spec/#reserved-field-ids) only reserves **`_`-prefixed** names for metad
   ata columns (`_file`, `_pos`, `_partition`, `_spec_id`, `_deleted`). The bare name `pos` is a valid data column name — the Java reference implementation does not treat it as a metadata column and reads it normally.
   
   ## Actual behavior (end-to-end)
   
   With a real table whose schema is `chrom: String (id=1)`, `pos: Int (id=2)`:
   
   ```text
   SELECT chrom FROM t   -> OK
   SELECT pos    FROM t   -> External error: Unexpected => "field not found"
   SELECT *      FROM t   -> External error: Unexpected => "field not found"   (pos is in the projection)
   ```
   
   The catalog schema is correct (`id=2 name=pos type=Int`), so this is purely a read-path problem. Workaround: rename the column before writing (`pos` -> `position`, `chro
   m_pos`, etc.).
   
   ## Root cause
   
   `crates/iceberg/src/metadata_columns.rs` defines the bare reserved names — these describe the columns **inside** a position-delete file, not metadata columns of a data t
   able:
   
   ```rust
   pub const RESERVED_COL_NAME_DELETE_FILE_POS: &str  = "pos";        // no underscore
   pub const RESERVED_COL_NAME_DELETE_FILE_PATH: &str = "file_path";  // no underscore
   ```
   
   `is_metadata_column_name()` (via `get_metadata_field_id`) returns `Ok` for both.
   
   `crates/iceberg/src/scan/mod.rs` applies that check to **every projected column of a data-table scan**, with metadata-first precedence and `continue` (no fallback to the
    data schema):
   
   ```rust
   for column_name in column_names.iter() {
       if is_metadata_column_name(column_name) {
           field_ids.push(get_metadata_field_id(column_name)?); // pos -> RESERVED_FIELD_ID_DELETE_FILE_POS
           continue;                                            // never reaches field_id_by_name
       }
       let field_id = schema.field_id_by_name(column_name)?;    // real data column never consulted
       ...
   }
   ```
   
   So even though the table has `pos` as field id `2`, the name `pos` is intercepted by the metadata branch before the data-schema lookup, and the scanner goes looking for 
   `RESERVED_FIELD_ID_DELETE_FILE_POS` in the Parquet file, which has no such column → `field not found`.
   
   ### Why metadata-first precedence is usually fine — and why it breaks here
   
   The data-table metadata columns (`_file`, `_pos`, `_partition`, `_spec_id`, `_deleted`) are **virtual**: they don't exist in the data schema or the Parquet file, they ar
   e injected by the scanner via a reserved field id. The "check metadata first, then `continue`" rule is how those virtual columns get materialized. The spec makes this sa
   fe by restricting metadata names to a `_` prefix, which **cannot collide with user data columns**. The bug is that `pos`/`file_path` (delete-file internal schema columns
   ) were added to the same name set consulted by data-table scans, but **without** the `_` prefix — so they can and do collide with legitimate data column names, and the m
   etadata-first precedence silently shadows them.
   
   | name | spec role | `_`-prefixed? | metadata-first safe? |
   |------|-----------|---------------|----------------------|
   | `_file` `_pos` `_partition` | virtual metadata columns | yes | yes (can't collide) |
   | `pos` `file_path` | delete-file **internal** schema columns | no | **no** (collides with user data) |
   
   This is a real-world problem for bioinformatics: VCF `POS` is one of the most common column names (oxbow emits it as `pos`), so any VCF → Iceberg pipeline hits this.
   
   ## Suggested fix
   
   Separate the two concepts so data-table scans only match the spec's `_`-prefixed metadata names:
   
   - Keep `RESERVED_COL_NAME_POS = "_pos"` (and `_file`, `_partition`, `_spec_id`, `_deleted`) in the set consulted by the scan path's `is_metadata_column_name` check.
   - Do **not** let `RESERVED_COL_NAME_DELETE_FILE_POS = "pos"` / `RESERVED_COL_NAME_DELETE_FILE_PATH = "file_path"` satisfy the scan-path metadata check — they should only
    be used when reading a delete file's own schema.
   
   (Alternative: in the metadata branch, if the name also exists as a real data column in the current table schema, prefer the data column's field id. But restricting the s
   et to `_`-prefixed names is cleaner and restores the spec's collision-safety guarantee.)
   
   ## Environment
   
   - iceberg-rust: git rev `713e201` (workspace `crates/iceberg`)
   - observed via the `iceberg-datafusion` table provider + REST catalog, but the bug is in core `scan` + `metadata_columns`
   
   Happy to open a PR for the suggested fix if the approach sounds right.
   
   ### To Reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Willingness to contribute
   
   None


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found" [iceberg-rust]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #2837:
URL: https://github.com/apache/iceberg-rust/issues/2837#issuecomment-5042187274

   Thanks, that's really helpful context, and good to know the minimal fix already unblocks you.
   
   On the overlap: #2746 adds `_pos` as a virtual metadata column, which is a different name from the bare `pos` this fixes. They line up rather than collide. The guard here keys off the `_` prefix, so `_pos` stays a metadata column while a plain `pos` goes back to being a normal data column — if anything this keeps the two from being confused. I'll bring the merged `_pos` work into this branch so it sits on current main.
   
   Your `pos_col` remapping is exactly the kind of per-table special-casing this should let you drop, so thanks for laying out the use case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found" [iceberg-rust]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY closed issue #2837: Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found"
URL: https://github.com/apache/iceberg-rust/issues/2837


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found" [iceberg-rust]

Posted by "wjixiang (via GitHub)" <gi...@apache.org>.
wjixiang commented on issue #2837:
URL: https://github.com/apache/iceberg-rust/issues/2837#issuecomment-5037058776

   Hi @LuciferYang,
   
   Thanks so much for the detailed breakdown and for surfacing both the minimal fix and the deeper cleanup option — really appreciate the thoroughness.
   
   As someone who's just using Iceberg and not deeply familiar with the internals, I don't feel qualified to weigh in strongly on which approach is technically better. That said, from a user perspective, your current minimal fix would already be a huge help — it would unblock my use case completely.
   
   For some context: I'm building a query engine on top of Datafusion for parsing genomic data, and the pos column collision is exactly what I'm running into. My current workaround is to remap pos to pos_col at the query layer, which works but feels brittle and forces me to special-case Iceberg tables. 
   
   One thing I noticed — there's parallel work going on in #2607/#2746 to add _pos as an official metadata column. I'm not sure how that intersects with your PR, but I figured it's worth mentioning in case it influences your approach.
   
   Either way, I'm happy to go with whatever you and the maintainers think is best. Just wanted to share my use case and say that I'd really appreciate having this fix landed.
   
   Thanks again for picking this up!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [I] Data column named `pos` (or `file_path`) is misread as a reserved metadata column during scan -> "field not found" [iceberg-rust]

Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on issue #2837:
URL: https://github.com/apache/iceberg-rust/issues/2837#issuecomment-5034463205

   I hit this one with a VCF → Iceberg pipeline (a column named `pos`), dug into the root cause, and opened #2865 for it.
   
   The fix guards `is_metadata_column_name` with a `starts_with('_')` check, so the two bare delete-file column names `pos`/`file_path` are no longer treated as projectable data-table metadata columns, while `get_metadata_field_id` keeps mapping them to their reserved ids for the delete read path. This matches the spec's `_`-prefix reservation and what the Java reader does.
   
   One design choice I'd like your take on: this is the minimal fix, on the scan consumer side. The root cause is that `pos`/`file_path` sit in the shared name→id map at all, so `get_metadata_field_id("pos").is_ok()` still returns true and the two public predicates now disagree. Removing those two names from the name→id map would be the deeper fix and looks safe in-repo (the delete read path uses the id→field direction, which stays untouched), but it changes a second public function's behavior, so I left it out. I noted the tradeoff in the PR — happy to go deeper if you'd prefer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org