You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2026/09/25 02:19:02 UTC
[I] Spark: Add ResolveBranch test coverage for path identifiers [iceberg]
manuzhang opened a new issue, #18256:
URL: https://github.com/apache/iceberg/issues/18256
### Feature Request / Improvement
`ResolveBranch` pins the target branch during analysis and rewrites the relation identifier so later refreshes still point at the branch. For path-based tables it has two dedicated cases:
```scala
case path: PathIdentifier if path.location.contains("#") =>
new PathIdentifier(path.location + "," + branchSelector)
case path: PathIdentifier =>
new PathIdentifier(path.location + "#" + branchSelector)
case _ =>
Identifier.of(ident.namespace :+ ident.name, branchSelector)
```
Neither `PathIdentifier` case has any test coverage. Nothing under `spark/*/spark-extensions/src/test` references `ResolveBranch`, and the only existing tests that combine a path load with `SparkReadOptions.BRANCH` are negative ones in `TestSnapshotSelection` asserting `"Cannot time travel in branch"`. So the behavior that is actually exercised today is the catalog-identifier `case _`.
That leaves three things unverified for tables loaded by path:
1. Reading a branch through `.option("branch", ...)` resolves to a `SparkTable` pinned to that branch, with the identifier rewritten to `<location>#branch_<name>`.
2. When the location already carries a metadata selector, the branch selector is appended with a comma rather than a second `#`, giving `<location>#files,branch_<name>`.
3. Writing through `.option("branch", ...)` appends to the branch and leaves the main table unchanged.
The comma-vs-hash distinction in particular is easy to break in a refactor and would surface only as a confusing path-parsing error at refresh time.
Proposal: add a `TestResolveBranch` under `spark-extensions` covering the read, metadata-selector, and write paths against a `HadoopTables` table. This is test-only; no production behavior change is intended.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement independently
---
**AI Disclosure**
- Model: Claude Opus 5
- Platform/Tool: Claude Code
- Human Oversight: partially reviewed
- Prompt Summary: file an issue describing the missing `ResolveBranch` path-identifier test coverage in the Spark extensions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark: Add ResolveBranch test coverage for path identifiers [iceberg]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #18256:
URL: https://github.com/apache/iceberg/issues/18256#issuecomment-5836108510
PR opened: https://github.com/apache/iceberg/pull/18261
Added `TestResolveBranch` for both Spark 4.1 and 4.2 (identical files).
One finding worth a maintainer's eye, since it changes what these tests can assert: while writing them I found the premise in the issue only partly holds on current `main`.
- For a path load with `.option("branch", ...)`, `IcebergSource.catalogAndIdentifier` already builds `PathIdentifier(path + "#branch_x")` before analysis runs, so `ResolveBranch` returns the relation **unchanged** and its `PathIdentifier` cases never fire.
- `<location>#files` is a metadata-table load, where the branch is never determined (`branch == null`), so the comma-append case is unreachable that way too.
- The only route I could reproduce that actually reaches `case path: PathIdentifier` is a branch taken from the **session WAP branch** (`spark.wap.branch`).
The tests therefore assert the rewrite that is genuinely observable (`<location>#branch_test`) plus the metadata-selector and write cases with their actual behavior, rather than an assertion that would pass for the wrong reason. If the `path.location.contains("#")` case is intentionally defensive, it may be worth saying so in the code or dropping it; I'd appreciate a pointer if there is a public route I missed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Spark: Add ResolveBranch test coverage for path identifiers [iceberg]
Posted by "zhang-arvin (via GitHub)" <gi...@apache.org>.
zhang-arvin commented on issue #18256:
URL: https://github.com/apache/iceberg/issues/18256#issuecomment-5835493895
@zhang-arvin here — I'd like to take this one.
Confirmed the gap: `ResolveBranch` rewrites the relation identifier for path-based tables in two cases (`location.contains("#")` → append `,branch_<name>`, otherwise `#branch_<name>`), and grep over `spark/*/spark-extensions/src/test` shows no test referencing `ResolveBranch`. The only path+branch tests today are the negative assertions in `TestSnapshotSelection` (`"Cannot time travel in branch"`), so the `case _` catalog path is the only one exercised.
Plan:
1. Add a `TestResolveBranch` (or extend `TestSnapshotSelection`) covering read via `.option("branch", ...)` on a path-loaded table, asserting the identifier is rewritten to `<location>#branch_<name>`.
2. Cover the second case where the location already carries a metadata selector, asserting the branch selector is appended with a comma → `<location>#files,branch_<name>`.
3. Cover the write path, asserting the append lands on the branch and `main` is untouched.
Will keep it test-only, no production behavior change. Please assign this to me if convenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org