You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "vishnuprakaz (via GitHub)" <gi...@apache.org> on 2026/07/15 13:13:13 UTC
[PR] perf(avro): cache Avro-to-Iceberg schema conversion [iceberg-python]
vishnuprakaz opened a new pull request, #3663:
URL: https://github.com/apache/iceberg-python/pull/3663
Closes #3662
# Rationale for this change
`AvroFileHeader.get_schema()` re-parses the embedded Avro schema and converts it to an Iceberg schema on every file
open. All manifests in a table share the same schema, so during scan planning we redo that identical conversion
once per manifest for no reason.
Caching it (keyed on the schema string) means each distinct schema is converted only once.
I timed `scan().plan_files()` on unpartitioned tables of a few sizes, `main` vs this branch (median of 10 warm runs):
| manifests | before | after | speedup |
|---:|---:|---:|---:|
| 10 | 5.6 ms | 3.1 ms | 1.8x |
| 50 | 29.4 ms | 15.6 ms | 1.9x |
| 150 | 86.2 ms | 48.0 ms | 1.8x |
| 400 | 233.9 ms | 128.8 ms | 1.8x |
Roughly 1.8x throughout, and since the wasted work is per-manifest, larger tables save more. Results are identical.
This is a narrow, unpartitioned table; a partitioned one has a wider manifest schema and would gain at least as
much.
Uses `functools.lru_cache`, like the existing `_cached_resolve_s3_region` in `io/pyarrow.py`, so no new dependency.
`Schema` is immutable and already shared across scan threads, so sharing one cached instance is safe.
## Are these changes tested?
Yes. A unit test in `tests/avro/test_file.py` checks the cached schema is correct, that the same schema string
returns the same object, and that a different string returns a different schema. There's also a benchmark in
`tests/benchmark/test_avro_schema_cache_benchmark.py` (run with `-m benchmark`) for the numbers above. Existing
avro/table/manifest tests pass.
## Are there any user-facing changes?
No, same behavior, just faster scan planning on tables with many manifests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(avro): cache Avro-to-Iceberg schema conversion [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3663:
URL: https://github.com/apache/iceberg-python/pull/3663#issuecomment-5383315417
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(avro): cache Avro-to-Iceberg schema conversion [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3663:
URL: https://github.com/apache/iceberg-python/pull/3663#issuecomment-5299408853
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(avro): cache Avro-to-Iceberg schema conversion [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3663: perf(avro): cache Avro-to-Iceberg schema conversion
URL: https://github.com/apache/iceberg-python/pull/3663
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org