You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "SergeyTimofeev1989 (via GitHub)" <gi...@apache.org> on 2026/10/22 13:24:15 UTC
[I] Clarify that write.*.isolation-level properties do not apply to append/INSERT [iceberg]
SergeyTimofeev1989 opened a new issue, #18608:
URL: https://github.com/apache/iceberg/issues/18608
### Problem / Motivation
The Configuration page documents three isolation-level table properties:
- `write.delete.isolation-level`
- `write.update.isolation-level`
- `write.merge.isolation-level`
There is no explicit note that these properties are scoped **only** to row-level operations
(DELETE, UPDATE, MERGE) and have **no effect** on append/INSERT operations. As a result, users
reasonably assume that setting one of these properties can reduce commit conflicts or commit
duration for append-heavy workloads. This is a common source of misunderstanding when
diagnosing 409 `CommitFailedException` responses from a REST catalog.
### Real-world scenario that motivated this request
We run 5 Iceberg tables, each written by ~30 independent Flink append jobs with a 1-minute
checkpoint interval (≈27 commits/minute per table), committed through a REST catalog
(Lakekeeper). We observed frequent 409 responses from the catalog (`CommitFailedException`,
`assert-ref-snapshot-id` not satisfied), with commit retries.
Our team set `write.delete.isolation-level = snapshot` on these tables, expecting it to reduce
commit conflicts or commit duration. It had **no effect** on the append path, because:
1. The Flink append path commits via `AppendFiles` (`table.newAppend()`), which does **not**
invoke `validateNoConflictingData` or any data-conflict validation.
2. The `write.*.isolation-level` properties are read by Spark's DELETE / UPDATE / MERGE paths,
not by append.
3. A 409 from the REST catalog is a **metadata-pointer conflict**, resolved by retry regardless
of isolation level (see the "Concurrent write operations" / "Retry validation" sections on
the Reliability page).
### Proposed change
Add an explicit note to the documentation stating that:
- `write.delete.isolation-level`, `write.update.isolation-level`, and `write.merge.isolation-level`
apply only to row-level operations (DELETE, UPDATE, MERGE).
- Append/INSERT has **no isolation level**; concurrent append conflicts are resolved at the
metadata level via retry, and there is no data-conflict check to configure.
Suggested locations (either or both):
- **Configuration page** → Write properties section, next to the three isolation-level rows.
- **Reliability page** → "Concurrent write operations" / "Retry validation" subsection, as a
short cross-reference clarifying that retry-based conflict resolution applies to append.
### Additional context
- Iceberg versions in use: 1.8.x (Spark maintenance) and 1.11.0 (Flink 2.1.2, `iceberg-flink-runtime-2.1`).
- Catalog: REST (Lakekeeper), S3 storage with remote signing.
- The confusion is not about a missing feature — it is about the docs not making the
append/row-level scoping of the `write.*.isolation-level` properties explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Clarify that write.*.isolation-level properties do not apply to append/INSERT [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #18608:
URL: https://github.com/apache/iceberg/issues/18608#issuecomment-6354735787
I'm really confused on the expectation here. Why would changing the isolation level matter for appends at all?
> Our team set write.delete.isolation-level = snapshot on these tables, expecting it to reduce
> commit conflicts or commit duration.
This doesn't make sense to me ... why would it do so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Clarify that write.*.isolation-level properties do not apply to append/INSERT [iceberg]
Posted by "SergeyTimofeev1989 (via GitHub)" <gi...@apache.org>.
SergeyTimofeev1989 commented on issue #18608:
URL: https://github.com/apache/iceberg/issues/18608#issuecomment-6363594573
> 2\. I'm really confused on the expectation here. Why would changing the isolation level matter for appends at all?
> > Our team set write.delete.isolation-level = snapshot on these tables, expecting it to reduce
> > commit conflicts or commit duration.
>
>
> This doesn't make sense to me ... why would it do so?
Fair question, and you're right: it doesn't, and it shouldn't. We're not asking for a behavior change, only for the docs to say this, because as written they led us to expect the opposite. Two things caused it:
1. "Isolation" means two different things in the docs. The Reliability page lists serializable isolation as a table guarantee ("All table changes occur in a linear history of atomic table updates"). The Configuration page then offers `write.delete.isolation-level` with default `serializable` and the description "Isolation level for delete commands: serializable or snapshot". Read together, it looks like a table-level switch for that guarantee, which would cover appends too.
2. On our tables it did have an effect, just not on the side we expected. Flink appends and Spark DELETEs (tiering and TTL jobs) commit to the same tables concurrently. After switching to `snapshot`, the Spark DELETEs mostly stopped failing with `Found conflicting files`, but the Flink side didn't change at all: same 409s in production, same commit duration in our test environment. That's when we went to the code.
What would have saved us the detour is one sentence saying which side of a concurrent commit these properties apply to, for example:
> Applies only to the DELETE, UPDATE or MERGE command being committed: it controls whether data files added by concurrent commits (including appends) count as conflicts. It has no effect on how appends are committed; an append that loses a commit race is retried according to `commit.retry.*`.
Happy to open a PR with that wording. If you don't think it's worth it, feel free to close the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Clarify that write.*.isolation-level properties do not apply to append/INSERT [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on issue #18608:
URL: https://github.com/apache/iceberg/issues/18608#issuecomment-6421269906
I'm still not following the jump from "Isolation level for delete commands" to "this applies to appends." The property name and the description both scope it to deletes. Isolation level is a table-level guarantee about what a commit is allowed to have observed, and in that sense appends are serializable too. An append reads nothing, so no concurrent commit can invalidate it, which is why there is nothing for it to validate. Explaining that properly means writing a guide to serializability in Iceberg, and I think that's out of scope for the configuration table. If you're changing these properties, you need to understand the guarantees you're giving up.
I also think the proposed sentence is misleading:
> Applies only to the DELETE, UPDATE or MERGE command being committed: it controls whether data files added by concurrent commits (including appends) count as conflicts. It has no effect on how appends are committed; an append that loses a commit race is retried according to commit.retry.*.
Every commit that loses a race is retried according to `commit.retry.*` (This includes DELETE, UPDATE and MERGE.) On retry, the operation re-validates against the new table state, and the isolation level decides whether that validation passes:
*serializable*: no concurrent commit added or removed data that matches what this operation read
*snapshot*: no concurrent commit removed or deleted data that matches what this operation read
Appends have no read set, so they always pass.
For your workload, the 409s on the Flink side are plain commit contention. Thirty writers checkpointing every minute against one table means roughly 27 commits a minute competing for the same metadata pointer. Isolation level can't help with that. Fewer commits (longer checkpoint intervals, or fewer writers per table) or more generous commit.retry.* settings can.
So my vote would be no change to the documentation here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org