[PR] perf(compaction): skip partition work for non-candidate files [iceberg-go]

["fallintoplace (via GitHub)" <gi...@apache.org>]

fallintoplace opened a new pull request, #2276:
URL: https://github.com/apache/iceberg-go/pull/2276

   ## What
   Skip partition-key work for files that will not be compacted.
   
   ## Why
   Large scans can contain many ineligible files. Building their keys and empty buckets uses extra time and memory.
   
   ## Implementation
   - Classify files before encoding partition keys.
   - Store candidate partitions in an indexed slice.
   - Keep candidate order within each partition.
   - Cover skipped files in candidate and empty partitions.
   
   ## Benchmark
   End-to-end `PlanCompaction`. Go 1.26.3, Apple M1 Pro, darwin/arm64, `GOMAXPROCS=1`; benchstat median of 10 runs, 500 ms each. Base: `a4fb4d0`.
   
   | Files / partitions | Candidates | Before | After | Less time | Allocs/op |
   | --- | ---: | ---: | ---: | ---: | ---: |
   | 20k / 256 | 0% | 8.067 ms | 0.329 ms | 95.9% | 60,020 → 0 |
   | 20k / 256 | 10% | 8.551 ms | 1.727 ms | 79.8% | 62,534 → 8,533 |
   | 20k / 256 | 50% | 10.563 ms | 6.265 ms | 40.7% | 63,937 → 33,936 |
   | 20k / 256 | 100% | 14.62 ms | 11.88 ms | 18.7% | 66,430 → 66,432 |
   | 100k / 1,024 | 10% | 39.931 ms | 8.793 ms | 78.0% | 311,475 → 41,476 |
   | 100k / 1,024 | 100% | 73.36 ms | 56.06 ms | 23.6% | 326,667 → 326,668 |
   
   Reproduce with:
   
   ```sh
   GOMAXPROCS=1 go test ./table/compaction -run '^$' -bench '^BenchmarkPlanCompaction$' -benchmem -benchtime=500ms -count=10
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [PR] perf(compaction): skip partition work for non-candidate files [iceberg-go]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade merged PR #2276:
URL: https://github.com/apache/iceberg-go/pull/2276


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org