You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "manuzhang (via GitHub)" <gi...@apache.org> on 2026/03/16 03:40:48 UTC
[PR] Core: Use ArrayList for manifest list materialization [iceberg]
manuzhang opened a new pull request, #15640:
URL: https://github.com/apache/iceberg/pull/15640
For lower memory overhead and better locality. It's used for `allManifests` in `BaseSnapshot`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra commented on PR #15640:
URL: https://github.com/apache/iceberg/pull/15640#issuecomment-4068073157
please run `ManifestReadBenchmark` and share before/after results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #15640:
URL: https://github.com/apache/iceberg/pull/15640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "Baunsgaard (via GitHub)" <gi...@apache.org>.
Baunsgaard commented on PR #15640:
URL: https://github.com/apache/iceberg/pull/15640#issuecomment-4261376094
Just to help out a bit I ran the benchmark. While there are no practical difference end-to-end since other things dominate, it is a minor but valid improvement.
## ManifestReadBenchmark Results
OpenJDK 17.0.18, Ubuntu 22.04, warmup=5, measure=10 iterations.
### End-to-end: read manifest list + iterate all data files
| Size | LinkedList | ArrayList | Ratio |
|-----------------------------|-------------|-------------|-------|
| 5 manifests × 100 rows | 9.70 ms | 9.48 ms | 1.02x |
| 50 manifests × 1K rows | 316.5 ms | 317.8 ms | 1.00x |
| 200 manifests × 10K rows | 11,827 ms | 11,820 ms | 1.00x |
### Materialization only (ManifestLists.read)
| Size | LinkedList | ArrayList | Ratio |
|---------------|------------|-----------|-------|
| 5 manifests | 868 us | 841 us | 1.03x |
| 50 manifests | 864 us | 835 us | 1.04x |
| 200 manifests | 1.14 ms | 1.15 ms | 0.99x |
## Extra specific measurements:
### Iteration after materialization
| Size | LinkedList | ArrayList | Ratio |
|---------------|------------|-----------|-----------|
| 5 manifests | 1.9 us | 730 ns | **2.60x** |
| 50 manifests | 12.2 us | 6.9 us | **1.76x** |
| 200 manifests | 25.9 us | 23.2 us | 1.12x |
### addAll into ArrayList (FastAppend.apply pattern)
| Size | LinkedList | ArrayList | Ratio |
|---------------|------------|-----------|-----------|
| 5 manifests | 847 ns | 546 ns | **1.55x** |
| 50 manifests | 1.3 us | 486 ns | **2.59x** |
| 200 manifests | 3.8 us | 523 ns | **7.33x** |
### Full pipeline: read + filter + ImmutableList.copyOf
| Size | LinkedList | ArrayList | Ratio |
|---------------|------------|-----------|-------|
| 5 manifests | 409 us | 422 us | 0.97x |
| 50 manifests | 387 us | 390 us | 0.99x |
| 200 manifests | 566 us | 581 us | 0.97x |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #15640:
URL: https://github.com/apache/iceberg/pull/15640#issuecomment-4265170017
@Baunsgaard Thanks for help on the benchmark. I have similar results locally. It's up to the community whether this is worth a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15640:
URL: https://github.com/apache/iceberg/pull/15640#issuecomment-4256574887
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core: Use ArrayList for manifest list materialization [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #15640:
URL: https://github.com/apache/iceberg/pull/15640#issuecomment-4473191986
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org