You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "fallintoplace (via GitHub)" <gi...@apache.org> on 2026/08/29 13:21:35 UTC
[PR] perf(table): parallelize non-bulk PurgeFiles deletion [iceberg-go]
fallintoplace opened a new pull request, #1973:
URL: https://github.com/apache/iceberg-go/pull/1973
## Summary
- **Parallelizes non-bulk `PurgeFiles` deletes** with bounded workers.
- Keeps `BulkRemovableIO` as the first choice.
- Keeps missing-file deletes successful.
- Collects independent failures in deterministic input order.
- Stops queued work when the context is canceled.
## Benchmark
Simulated 100µs per-file deletion:
- **1,000 files:** 120ms with 1 worker, 30ms with 4 workers, 7.7ms with 16 workers.
- **10,000 files:** 1.22s with 1 worker, 416ms with 4 workers, 76ms with 16 workers.
## Tests
- `go test ./table -count=1`
- `go test -race ./table -count=1`
- `go vet ./...`
- `golangci-lint run --timeout=10m`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): parallelize non-bulk PurgeFiles deletion [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #1973:
URL: https://github.com/apache/iceberg-go/pull/1973#discussion_r3903436370
##########
table/orphan_cleanup_test.go:
##########
@@ -1616,6 +1622,194 @@ func (m mockFileInfo) ModTime() time.Time { return m.modTime }
func (m mockFileInfo) IsDir() bool { return m.mode.IsDir() }
func (m mockFileInfo) Sys() any { return nil }
+type purgeDeleteTrackingIO struct {
+ mockListableIO
+
+ targetActive int
+ reached chan struct{}
+ release chan struct{}
+
+ mu sync.Mutex
+ active int
+ maxActive int
+ removed []string
+ reachedOnce sync.Once
+ releaseOnce sync.Once
+}
+
+func (m *purgeDeleteTrackingIO) Remove(name string) error {
+ m.mu.Lock()
+ m.active++
+ if m.active > m.maxActive {
+ m.maxActive = m.active
+ }
+ if m.targetActive > 0 && m.active >= m.targetActive {
+ m.reachedOnce.Do(func() { close(m.reached) })
+ }
+ m.mu.Unlock()
+
+ if m.targetActive > 0 {
+ <-m.release
+ }
+
+ m.mu.Lock()
+ m.active--
+ m.removed = append(m.removed, name)
+ m.mu.Unlock()
+
+ return nil
+}
+
+func (m *purgeDeleteTrackingIO) MaxActive() int {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+
+ return m.maxActive
+}
+
+func (m *purgeDeleteTrackingIO) Release() {
+ m.releaseOnce.Do(func() { close(m.release) })
+}
+
+func TestDeleteFilesParallelCollectsPurgeErrors(t *testing.T) {
+ const (
+ firstPath = "s3://bucket/table/first.parquet"
+ missingPath = "s3://bucket/table/missing.parquet"
+ slowPath = "s3://bucket/table/slow.parquet"
+ fastPath = "s3://bucket/table/fast.parquet"
+ )
+
+ slowErr := errors.New("slow removal failed")
+ fastErr := errors.New("fast removal failed")
+ files := []string{firstPath, missingPath, slowPath, fastPath}
+ var mu sync.Mutex
+ calls := make(map[string]int)
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 4,
+ func(path string) error {
+ mu.Lock()
+ calls[path]++
+ mu.Unlock()
+
+ var err error
+ switch path {
+ case missingPath:
+ err = stdfs.ErrNotExist
+ case slowPath:
+ time.Sleep(10 * time.Millisecond)
+
+ err = slowErr
+ case fastPath:
+ err = fastErr
+ }
+ if os.IsNotExist(err) {
+ return nil
+ }
+
+ return err
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ assert.Equal(t, []string{firstPath, missingPath}, deleted)
+ require.ErrorIs(t, err, slowErr)
+ require.ErrorIs(t, err, fastErr)
+ assert.NotContains(t, err.Error(), missingPath)
+ assert.Less(t, strings.Index(err.Error(), slowPath), strings.Index(err.Error(), fastPath))
+
+ mu.Lock()
+ assert.Equal(t, map[string]int{
+ firstPath: 1,
+ missingPath: 1,
+ slowPath: 1,
+ fastPath: 1,
+ }, calls)
+ mu.Unlock()
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnCancellation(t *testing.T) {
Review Comment:
This only exercises the pre-cancelled case: ctx is already cancelled before the call, and with a single file `workers` is `min(max(4,1), 1) == 1`, so the sender's `case <-ctx.Done()` fires immediately and nothing ever lands on `jobs`. The post-receive guard (`if err := ctx.Err(); err != nil` inside the worker) never runs, so this would still pass if that guard were deleted.
I'd add a second case that actually exercises mid-dispatch: say 50 files with 4 workers, cancel from inside `deleteFunc` after a handful of calls complete, then assert the deleted count is well short of 50 and `err` carries `context.Canceled`. That's the path that proves queued work actually stops. wdyt?
##########
table/orphan_cleanup_bench_test.go:
##########
@@ -124,3 +124,45 @@ func (fs *benchmarkDelayIO) Open(name string) (iceio.File, error) {
return f, nil
}
+
+// BenchmarkPurgeFilesNonBulkDeletion measures the bounded fallback used when
+// the filesystem does not implement BulkRemovableIO. The delay models the
+// round trip to a remote object store; the zero-delay cases show the worker
+// pool overhead on local-style deletes.
+func BenchmarkPurgeFilesNonBulkDeletion(b *testing.B) {
+ for _, fileCount := range []int{100, 1_000, 10_000} {
+ files := make([]string, fileCount)
+ for i := range files {
+ files[i] = fmt.Sprintf("s3://bucket/table/data/file-%d.parquet", i)
+ }
+
+ for _, delay := range []time.Duration{0, 100 * time.Microsecond, time.Millisecond} {
+ for _, concurrency := range []int{1, 4, 16} {
+ b.Run(fmt.Sprintf("files=%d/delay=%s/concurrency=%d", fileCount, delay, concurrency), func(b *testing.B) {
+ deleteFunc := func(string) error {
+ time.Sleep(delay)
+
+ return nil
+ }
+
+ b.ReportAllocs()
+ b.ReportMetric(float64(fileCount), "files/op")
+ b.ResetTimer()
+ for b.Loop() {
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ concurrency,
+ deleteFunc,
+ func(string, error) error { return nil },
+ )
+ if err != nil {
+ b.Fatal(err)
+ }
+ orphanCleanupBenchmarkSink = deleted[len(deleted)-1]
Review Comment:
`deleted[len(deleted)-1]` is safe only because this bench's `deleteFunc` always succeeds, so every index ends up in `deleted`. Worth noting the underlying shape though: if a delete fails and `wrapError` returns nil (as it does here), that index lands in neither `deleted` nor `deleteErrors` and silently drops from both return slices. Extend this bench to model partial failures and `deleted` could be empty, so this indexes out of bounds.
Simplest guard is to make `wrapError` actually wrap (`func(_ string, err error) error { return err }`); that keeps the sink safe and surfaces any failure through `b.Fatal`. wdyt?
##########
table/orphan_cleanup.go:
##########
@@ -1334,15 +1382,23 @@ func (t Table) PurgeFiles(ctx context.Context) error {
errs = append(errs, fmt.Errorf("bulk deletion failed: %w", bulkErr))
}
} else {
- for _, file := range files {
- if err := ctx.Err(); err != nil {
- errs = append(errs, err)
+ _, removeErr := deleteFilesParallel(
+ ctx,
+ files,
+ runtime.GOMAXPROCS(0),
Review Comment:
`runtime.GOMAXPROCS(0)` is a CPU-count heuristic, but what we're parallelizing here is object-store round trips, not CPU work. On a typical cloud box that caps us at 2-8 concurrent deletes, and since throughput is roughly concurrency over latency, 4 workers at ~100ms/op is ~40 files/sec. That's a real win over sequential, but object-store clients usually run 32-64 in flight, so we're leaving close to an order of magnitude on the table.
The orphan-cleanup path already exposes `WithCleanupMaxConcurrency`; `PurgeFiles` has no knob at all. I'd either bump the default to something I/O-appropriate (32 is defensible) or add a `WithPurgeMaxConcurrency` option mirroring the cleanup one, and note in the doc that GOMAXPROCS is a CPU default that wants raising for remote stores. wdyt?
##########
table/orphan_cleanup.go:
##########
@@ -740,55 +758,85 @@ func deleteFilesSequential(fs iceio.IO, orphanFiles []string, cfg *orphanCleanup
return deletedFiles, result
}
-func deleteFilesParallel(fs iceio.IO, orphanFiles []string, cfg *orphanCleanupConfig) ([]string, error) {
- deleteFunc := fs.Remove
- if cfg.deleteFunc != nil {
- deleteFunc = cfg.deleteFunc
+func deleteFilesParallel(
+ ctx context.Context,
+ files []string,
+ maxConcurrency int,
+ deleteFunc func(string) error,
+ wrapError func(string, error) error,
+) ([]string, error) {
+ workers := min(max(maxConcurrency, 1), len(files))
+ jobs := make(chan int)
+ deleted := make([]bool, len(files))
+ deleteErrors := make([]error, len(files))
+
+ var cancellationErr error
+ var cancellationOnce sync.Once
+ recordCancellation := func(err error) {
+ cancellationOnce.Do(func() {
+ cancellationErr = err
+ })
}
- in := make(chan string, cfg.maxConcurrency)
- out := make(chan string, cfg.maxConcurrency)
- errList := make([][]error, cfg.maxConcurrency)
-
- go func() {
- defer close(in)
- for _, file := range orphanFiles {
- in <- file
- }
- }()
-
var wg sync.WaitGroup
- wg.Add(cfg.maxConcurrency)
- for i := range cfg.maxConcurrency {
- go func(workerID int) {
+ wg.Add(workers)
+ for range workers {
+ go func() {
defer wg.Done()
- for file := range in {
- if err := deleteFunc(file); err != nil {
- errList[workerID] = append(errList[workerID], fmt.Errorf("failed to delete orphan file %s: %w", file, err))
- } else {
- out <- file
+ for {
+ select {
+ case <-ctx.Done():
+ recordCancellation(ctx.Err())
+
+ return
+ case index, ok := <-jobs:
+ if !ok {
+ return
+ }
+ if err := ctx.Err(); err != nil {
Review Comment:
The outer `case <-ctx.Done()` and this inner `ctx.Err()` check look redundant but both are load-bearing: when a job and `ctx.Done()` are both ready, `select` picks at random, so this inner check is what stops a worker from doing real I/O after cancellation when the random pick happened to land on `jobs`. Correct and known pattern, just subtle. A one-line comment here would save the next maintainer from re-deriving it.
##########
table/orphan_cleanup.go:
##########
@@ -740,55 +758,85 @@ func deleteFilesSequential(fs iceio.IO, orphanFiles []string, cfg *orphanCleanup
return deletedFiles, result
}
-func deleteFilesParallel(fs iceio.IO, orphanFiles []string, cfg *orphanCleanupConfig) ([]string, error) {
- deleteFunc := fs.Remove
- if cfg.deleteFunc != nil {
- deleteFunc = cfg.deleteFunc
+func deleteFilesParallel(
+ ctx context.Context,
+ files []string,
+ maxConcurrency int,
+ deleteFunc func(string) error,
+ wrapError func(string, error) error,
+) ([]string, error) {
+ workers := min(max(maxConcurrency, 1), len(files))
+ jobs := make(chan int)
+ deleted := make([]bool, len(files))
+ deleteErrors := make([]error, len(files))
+
+ var cancellationErr error
+ var cancellationOnce sync.Once
+ recordCancellation := func(err error) {
+ cancellationOnce.Do(func() {
+ cancellationErr = err
+ })
}
- in := make(chan string, cfg.maxConcurrency)
- out := make(chan string, cfg.maxConcurrency)
- errList := make([][]error, cfg.maxConcurrency)
-
- go func() {
- defer close(in)
- for _, file := range orphanFiles {
- in <- file
- }
- }()
-
var wg sync.WaitGroup
- wg.Add(cfg.maxConcurrency)
- for i := range cfg.maxConcurrency {
- go func(workerID int) {
+ wg.Add(workers)
+ for range workers {
+ go func() {
defer wg.Done()
- for file := range in {
- if err := deleteFunc(file); err != nil {
- errList[workerID] = append(errList[workerID], fmt.Errorf("failed to delete orphan file %s: %w", file, err))
- } else {
- out <- file
+ for {
+ select {
+ case <-ctx.Done():
+ recordCancellation(ctx.Err())
+
+ return
+ case index, ok := <-jobs:
+ if !ok {
+ return
+ }
+ if err := ctx.Err(); err != nil {
+ recordCancellation(err)
+
+ return
+ }
+
+ if err := deleteFunc(files[index]); err != nil {
+ deleteErrors[index] = wrapError(files[index], err)
+ } else {
+ deleted[index] = true
+ }
}
}
- }(i)
+ }()
}
- go func() {
- wg.Wait()
- close(out)
- }()
+send:
+ for index := range files {
+ select {
+ case jobs <- index:
+ case <-ctx.Done():
+ recordCancellation(ctx.Err())
- deletedFiles := make([]string, 0, len(orphanFiles))
- for file := range out {
- deletedFiles = append(deletedFiles, file)
+ break send
+ }
}
+ close(jobs)
+ wg.Wait()
- var allErrors []error
- for _, workerErrors := range errList {
- allErrors = append(allErrors, workerErrors...)
+ deletedFiles := make([]string, 0, len(files))
+ allErrors := make([]error, 0)
Review Comment:
Small one: `var allErrors []error` reads cleaner here. `errors.Join` treats nil and empty-non-nil the same, so the explicit `make([]error, 0)` signals an intentional non-nil-empty that doesn't actually matter, and it trips the perfsprint-style lints.
##########
table/orphan_cleanup.go:
##########
@@ -729,6 +742,11 @@ func deleteFilesSequential(fs iceio.IO, orphanFiles []string, cfg *orphanCleanup
var result error
for _, file := range orphanFiles {
+ if err := ctx.Err(); err != nil {
+ result = errors.Join(result, err)
Review Comment:
Minor asymmetry with the parallel path: here the context error gets `errors.Join`'d into `result` inline, mid-chain with any delete failures, whereas `deleteFilesParallel` appends `cancellationErr` last. Both share the same caller in `deleteFiles`, so depending only on how `maxConcurrency` slices the same input, callers get structurally different error nesting. `errors.Is(context.Canceled)` works either way, but string and unwrap inspection differ.
I'd accumulate the context error separately here and join it last too, so the two paths produce the same shape. While we're at it, a line in the `PurgeFiles` doc noting non-bulk errors now come back as a single joined value would help, since the old path returned per-file errors directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): parallelize non-bulk PurgeFiles deletion [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1973:
URL: https://github.com/apache/iceberg-go/pull/1973#discussion_r3960399516
##########
table/orphan_cleanup.go:
##########
@@ -712,23 +714,44 @@ func deleteFiles(ctx context.Context, fs iceio.IO, orphanFiles []string, cfg *or
}
}
- if cfg.maxConcurrency == 1 {
- return deleteFilesSequential(fs, orphanFiles, cfg)
+ if cfg.maxConcurrency <= 1 {
+ return deleteFilesSequential(ctx, fs, orphanFiles, cfg)
+ }
+
+ deleteFunc := fs.Remove
+ if cfg.deleteFunc != nil {
+ deleteFunc = cfg.deleteFunc
}
- return deleteFilesParallel(fs, orphanFiles, cfg)
+ return deleteFilesParallel(
+ ctx,
+ orphanFiles,
+ cfg.maxConcurrency,
+ deleteFunc,
+ func(file string, err error) error {
+ return fmt.Errorf("failed to delete orphan file %s: %w", file, err)
+ },
+ )
}
-func deleteFilesSequential(fs iceio.IO, orphanFiles []string, cfg *orphanCleanupConfig) ([]string, error) {
+func deleteFilesSequential(ctx context.Context, fs iceio.IO, orphanFiles []string, cfg *orphanCleanupConfig) ([]string, error) {
var deletedFiles []string
deleteFunc := fs.Remove
if cfg.deleteFunc != nil {
deleteFunc = cfg.deleteFunc
}
- var result error
+ var (
+ result error
+ cancellationErr error
+ )
for _, file := range orphanFiles {
+ if err := ctx.Err(); err != nil {
Review Comment:
**minor** — New ctx cancellation guard in deleteFilesSequential is pinned by no test
This PR adds context cancellation to deleteFilesSequential (the function previously took no ctx at all). The guard is production-reachable: deleteFiles routes here when cfg.maxConcurrency <= 1, and maxConcurrency defaults to runtime.GOMAXPROCS(0) (orphan_cleanup.go:284), so single-CPU containers take this path by default; WithCleanupMaxConcurrency(1) also selects it. The parallel twin is now well pinned, but this one is not. Suggest a small test that cancels ctx after N sequential deletes and asserts the returned error is context.Canceled and that len(deletedFiles) < len(orphanFiles).
##########
table/orphan_cleanup_test.go:
##########
@@ -1616,6 +1624,243 @@ func (m mockFileInfo) ModTime() time.Time { return m.modTime }
func (m mockFileInfo) IsDir() bool { return m.mode.IsDir() }
func (m mockFileInfo) Sys() any { return nil }
+type purgeDeleteTrackingIO struct {
+ mockListableIO
+
+ release chan struct{}
+
+ mu sync.Mutex
+ active int
+ maxActive int
+ removed []string
+}
+
+func (m *purgeDeleteTrackingIO) Remove(name string) error {
+ m.mu.Lock()
+ m.active++
+ m.maxActive = max(m.maxActive, m.active)
+ m.mu.Unlock()
+
+ <-m.release
+
+ m.mu.Lock()
+ m.active--
+ m.removed = append(m.removed, name)
+ m.mu.Unlock()
+
+ return nil
+}
+
+func (m *purgeDeleteTrackingIO) MaxActive() int {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+
+ return m.maxActive
+}
+
+func TestDeleteFilesParallelCollectsPurgeErrors(t *testing.T) {
+ const (
+ firstPath = "s3://bucket/table/first.parquet"
+ missingPath = "s3://bucket/table/missing.parquet"
+ slowPath = "s3://bucket/table/slow.parquet"
+ fastPath = "s3://bucket/table/fast.parquet"
+ )
+
+ slowErr := errors.New("slow removal failed")
+ fastErr := errors.New("fast removal failed")
+ files := []string{firstPath, missingPath, slowPath, fastPath}
+ var mu sync.Mutex
+ calls := make(map[string]int)
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 4,
+ func(path string) error {
+ mu.Lock()
+ calls[path]++
+ mu.Unlock()
+
+ var err error
+ switch path {
+ case missingPath:
+ err = stdfs.ErrNotExist
+ case slowPath:
+ time.Sleep(10 * time.Millisecond)
+
+ err = slowErr
+ case fastPath:
+ err = fastErr
+ }
+ if os.IsNotExist(err) {
+ return nil
+ }
+
+ return err
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ assert.Equal(t, []string{firstPath, missingPath}, deleted)
+ require.ErrorIs(t, err, slowErr)
+ require.ErrorIs(t, err, fastErr)
+ assert.NotContains(t, err.Error(), missingPath)
+ assert.Less(t, strings.Index(err.Error(), slowPath), strings.Index(err.Error(), fastPath))
+
+ mu.Lock()
+ assert.Equal(t, map[string]int{
+ firstPath: 1,
+ missingPath: 1,
+ slowPath: 1,
+ fastPath: 1,
+ }, calls)
+ mu.Unlock()
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnCancellation(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+
+ called := false
+ _, err := deleteFilesParallel(
+ ctx,
+ []string{"s3://bucket/table/file.parquet"},
+ 4,
+ func(string) error {
+ called = true
+
+ return nil
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ require.ErrorIs(t, err, context.Canceled)
+ assert.False(t, called)
+}
+
+// gatedErrContext lets a test cancel after a job is received, before Err returns.
+type gatedErrContext struct {
+ context.Context
+ checks <-chan struct{}
+}
+
+func (c gatedErrContext) Err() error {
+ <-c.checks
+
+ return c.Context.Err()
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnMidFlightCancellation(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ const (
+ fileCount = 50
+ maxConcurrency = 4
+ )
+
+ files := make([]string, fileCount)
+ for i := range files {
+ files[i] = fmt.Sprintf("s3://bucket/table/file-%02d.parquet", i)
+ }
+
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ checks := make(chan struct{}, maxConcurrency)
+ for range maxConcurrency {
+ checks <- struct{}{}
+ }
+ var calls atomic.Int32
+ release := make(chan struct{})
+ var deleted []string
+ var err error
+ done := make(chan struct{})
+ go func() {
+ defer close(done)
+ deleted, err = deleteFilesParallel(
+ gatedErrContext{Context: ctx, checks: checks},
+ files,
+ maxConcurrency,
+ func(string) error {
+ calls.Add(1)
+ <-release
+
+ return nil
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+ }()
+
+ synctest.Wait()
+ assert.Equal(t, int32(maxConcurrency), calls.Load())
+
+ // Let one worker receive another job, then cancel while its Err check is paused.
+ release <- struct{}{}
+ synctest.Wait()
+ cancel()
+ close(checks)
+ close(release)
+ <-done
+
+ require.ErrorIs(t, err, context.Canceled)
+ assert.Equal(t, int32(maxConcurrency), calls.Load())
+ assert.Equal(t, files[:maxConcurrency], deleted)
+ })
+}
+
+func TestPurgeFilesDeletesNonBulkFilesConcurrently(t *testing.T) {
+ synctest.Test(t, func(t *testing.T) {
+ const maxWorkers = defaultPurgeMaxConcurrency
+ const fileCount = maxWorkers * 2
+ entries := make([]mockWalkEntry, 0, fileCount)
+ for i := range fileCount {
+ entries = append(entries, mockWalkEntry{
+ path: fmt.Sprintf("s3://bucket/table/data/file-%02d.parquet", i),
+ info: mockFileInfo{name: fmt.Sprintf("file-%02d.parquet", i)},
+ })
+ }
+
+ fsys := &purgeDeleteTrackingIO{
+ mockListableIO: mockListableIO{entries: entries},
+ release: make(chan struct{}),
+ }
+
+ meta, err := NewMetadata(
+ iceberg.NewSchema(0),
+ iceberg.UnpartitionedSpec,
+ UnsortedSortOrder,
+ "s3://bucket/table",
+ iceberg.Properties{},
+ )
+ require.NoError(t, err)
+ tbl := New(
+ Identifier{"db", "tbl"},
+ meta,
+ "s3://bucket/table/metadata/v1.metadata.json",
+ testFSF(fsys),
+ nil,
+ )
+
+ done := make(chan error, 1)
+ go func() { done <- tbl.PurgeFiles(context.Background()) }()
+
+ // Every worker remains blocked in Remove until the full pool is observable.
+ synctest.Wait()
+ assert.Greater(t, maxWorkers, 1)
+ assert.Equal(t, maxWorkers, fsys.MaxActive())
Review Comment:
**nit** — assert.Greater(t, maxWorkers, 1) is a compile-time-constant tautology
maxWorkers is 'const maxWorkers = defaultPurgeMaxConcurrency' (32), so this asserts 32 > 1 and exercises no code under test. The adjacent assert.Equal(t, maxWorkers, fsys.MaxActive()) already carries the real signal. Harmless, but it is exactly the kind of assertion that reads as coverage without providing any.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): parallelize non-bulk PurgeFiles deletion [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #1973:
URL: https://github.com/apache/iceberg-go/pull/1973#discussion_r3937964158
##########
table/orphan_cleanup_test.go:
##########
@@ -1616,6 +1623,256 @@ func (m mockFileInfo) ModTime() time.Time { return m.modTime }
func (m mockFileInfo) IsDir() bool { return m.mode.IsDir() }
func (m mockFileInfo) Sys() any { return nil }
+type purgeDeleteTrackingIO struct {
+ mockListableIO
+
+ targetActive int
+ reached chan struct{}
+ release chan struct{}
+
+ mu sync.Mutex
+ active int
+ maxActive int
+ removed []string
+ reachedOnce sync.Once
+ releaseOnce sync.Once
+}
+
+func (m *purgeDeleteTrackingIO) Remove(name string) error {
+ m.mu.Lock()
+ m.active++
+ if m.active > m.maxActive {
+ m.maxActive = m.active
+ }
+ if m.targetActive > 0 && m.active >= m.targetActive {
+ m.reachedOnce.Do(func() { close(m.reached) })
+ }
+ m.mu.Unlock()
+
+ if m.targetActive > 0 {
+ <-m.release
+ }
+
+ m.mu.Lock()
+ m.active--
+ m.removed = append(m.removed, name)
+ m.mu.Unlock()
+
+ return nil
+}
+
+func (m *purgeDeleteTrackingIO) MaxActive() int {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+
+ return m.maxActive
+}
+
+func (m *purgeDeleteTrackingIO) Release() {
+ m.releaseOnce.Do(func() { close(m.release) })
+}
+
+func TestDeleteFilesParallelCollectsPurgeErrors(t *testing.T) {
+ const (
+ firstPath = "s3://bucket/table/first.parquet"
+ missingPath = "s3://bucket/table/missing.parquet"
+ slowPath = "s3://bucket/table/slow.parquet"
+ fastPath = "s3://bucket/table/fast.parquet"
+ )
+
+ slowErr := errors.New("slow removal failed")
+ fastErr := errors.New("fast removal failed")
+ files := []string{firstPath, missingPath, slowPath, fastPath}
+ var mu sync.Mutex
+ calls := make(map[string]int)
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 4,
+ func(path string) error {
+ mu.Lock()
+ calls[path]++
+ mu.Unlock()
+
+ var err error
+ switch path {
+ case missingPath:
+ err = stdfs.ErrNotExist
+ case slowPath:
+ time.Sleep(10 * time.Millisecond)
+
+ err = slowErr
+ case fastPath:
+ err = fastErr
+ }
+ if os.IsNotExist(err) {
+ return nil
+ }
+
+ return err
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ assert.Equal(t, []string{firstPath, missingPath}, deleted)
+ require.ErrorIs(t, err, slowErr)
+ require.ErrorIs(t, err, fastErr)
+ assert.NotContains(t, err.Error(), missingPath)
+ assert.Less(t, strings.Index(err.Error(), slowPath), strings.Index(err.Error(), fastPath))
+
+ mu.Lock()
+ assert.Equal(t, map[string]int{
+ firstPath: 1,
+ missingPath: 1,
+ slowPath: 1,
+ fastPath: 1,
+ }, calls)
+ mu.Unlock()
+}
+
+func TestDeleteFilesParallelPreservesErrorWithNilWrapper(t *testing.T) {
+ deleteErr := errors.New("delete failed")
+ files := []string{
+ "s3://bucket/table/ok.parquet",
+ "s3://bucket/table/failed.parquet",
+ }
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 2,
+ func(path string) error {
+ if path == files[1] {
+ return deleteErr
+ }
+
+ return nil
+ },
+ func(string, error) error { return nil },
+ )
+
+ assert.Equal(t, []string{files[0]}, deleted)
+ require.ErrorIs(t, err, deleteErr)
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnCancellation(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+
+ called := false
+ _, err := deleteFilesParallel(
+ ctx,
+ []string{"s3://bucket/table/file.parquet"},
+ 4,
+ func(string) error {
+ called = true
+
+ return nil
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ require.ErrorIs(t, err, context.Canceled)
+ assert.False(t, called)
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnMidFlightCancellation(t *testing.T) {
+ const (
+ fileCount = 50
+ maxConcurrency = 4
+ )
+
+ files := make([]string, fileCount)
+ for i := range files {
+ files[i] = fmt.Sprintf("s3://bucket/table/file-%02d.parquet", i)
+ }
+
+ ctx, cancel := context.WithCancel(context.Background())
+ defer cancel()
+
+ var calls atomic.Int32
+ release := make(chan struct{})
+ deleted, err := deleteFilesParallel(
+ ctx,
+ files,
+ maxConcurrency,
+ func(string) error {
+ call := calls.Add(1)
+ if call == maxConcurrency {
+ cancel()
+ close(release)
+ } else if call < maxConcurrency {
+ <-release
+ }
+
+ return nil
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ require.ErrorIs(t, err, context.Canceled)
+ assert.Equal(t, int32(maxConcurrency), calls.Load())
+ assert.Less(t, len(deleted), fileCount)
+}
+
+func TestPurgeFilesDeletesNonBulkFilesConcurrently(t *testing.T) {
+ const maxWorkers = defaultPurgeMaxConcurrency
+ const fileCount = maxWorkers * 2
+ entries := make([]mockWalkEntry, 0, fileCount)
+ for i := range fileCount {
+ entries = append(entries, mockWalkEntry{
+ path: fmt.Sprintf("s3://bucket/table/data/file-%02d.parquet", i),
+ info: mockFileInfo{name: fmt.Sprintf("file-%02d.parquet", i)},
+ })
+ }
+
+ fsys := &purgeDeleteTrackingIO{
+ mockListableIO: mockListableIO{entries: entries},
+ targetActive: 2,
+ reached: make(chan struct{}),
+ release: make(chan struct{}),
+ }
+
+ meta, err := NewMetadata(
+ iceberg.NewSchema(0),
+ iceberg.UnpartitionedSpec,
+ UnsortedSortOrder,
+ "s3://bucket/table",
+ iceberg.Properties{},
+ )
+ require.NoError(t, err)
+ tbl := New(
+ Identifier{"db", "tbl"},
+ meta,
+ "s3://bucket/table/metadata/v1.metadata.json",
+ testFSF(fsys),
+ nil,
+ )
+
+ done := make(chan error, 1)
+ go func() { done <- tbl.PurgeFiles(context.Background()) }()
+
+ select {
+ case <-fsys.reached:
+ case <-time.After(5 * time.Second):
+ fsys.Release()
+ <-done
+ t.Fatal("non-bulk purge deletion did not reach two concurrent removals")
+ }
+ fsys.Release()
+
+ require.NoError(t, <-done)
+ assert.Greater(t, fsys.MaxActive(), 1)
+ assert.LessOrEqual(t, fsys.MaxActive(), maxWorkers)
+}
Review Comment:
**major** — assert.LessOrEqual(fsys.MaxActive(), maxWorkers) is vacuous - the concurrency cap is unverified
purgeDeleteTrackingIO is constructed with targetActive: 2 (line 1838), so the test releases the barrier as soon as two removals are concurrently active. The worker pool never saturates, so MaxActive() never approaches the cap and the upper-bound assertion can never fail. Bounded concurrency is the headline safety property of this PR and nothing pins it. Fix: set targetActive to defaultPurgeMaxConcurrency so all workers are held in flight, then assert MaxActive() == defaultPurgeMaxConcurrency exactly.
<details><summary>Evidence</summary>
```text
Replaced `workers := min(max(maxConcurrency, 1), len(files))` (orphan_cleanup.go:773) with `workers := len(files)` -- 64 workers against a documented cap of 32 -- and `go test -count=50 -run TestPurgeFilesDeletesNonBulkFilesConcurrently ./table/` => `ok ... 0.556s` (50/50 pass). A probe using a barrier that actually holds every worker observed the correct behaviour: 'PROBE-A bound=8 files=64 observedPeak=8', confirming the code is right and only the assertion is toothless.
```
</details>
##########
table/orphan_cleanup.go:
##########
@@ -737,58 +760,95 @@ func deleteFilesSequential(fs iceio.IO, orphanFiles []string, cfg *orphanCleanup
deletedFiles = append(deletedFiles, file)
}
- return deletedFiles, result
+ return deletedFiles, errors.Join(result, cancellationErr)
}
-func deleteFilesParallel(fs iceio.IO, orphanFiles []string, cfg *orphanCleanupConfig) ([]string, error) {
- deleteFunc := fs.Remove
- if cfg.deleteFunc != nil {
- deleteFunc = cfg.deleteFunc
+func deleteFilesParallel(
+ ctx context.Context,
+ files []string,
+ maxConcurrency int,
+ deleteFunc func(string) error,
+ wrapError func(string, error) error,
+) ([]string, error) {
+ workers := min(max(maxConcurrency, 1), len(files))
+ jobs := make(chan int)
+ deleted := make([]bool, len(files))
+ deleteErrors := make([]error, len(files))
+
+ var cancellationErr error
+ var cancellationOnce sync.Once
+ recordCancellation := func(err error) {
+ cancellationOnce.Do(func() {
+ cancellationErr = err
+ })
}
- in := make(chan string, cfg.maxConcurrency)
- out := make(chan string, cfg.maxConcurrency)
- errList := make([][]error, cfg.maxConcurrency)
-
- go func() {
- defer close(in)
- for _, file := range orphanFiles {
- in <- file
- }
- }()
-
var wg sync.WaitGroup
- wg.Add(cfg.maxConcurrency)
- for i := range cfg.maxConcurrency {
- go func(workerID int) {
+ wg.Add(workers)
+ for range workers {
+ go func() {
defer wg.Done()
- for file := range in {
- if err := deleteFunc(file); err != nil {
- errList[workerID] = append(errList[workerID], fmt.Errorf("failed to delete orphan file %s: %w", file, err))
- } else {
- out <- file
+ for {
+ select {
+ case <-ctx.Done():
+ recordCancellation(ctx.Err())
+
+ return
+ case index, ok := <-jobs:
+ if !ok {
+ return
+ }
+ // Both cases can be ready after cancellation, so check the context
+ // again before starting the deletion.
+ if err := ctx.Err(); err != nil {
+ recordCancellation(err)
+
+ return
+ }
+
+ if err := deleteFunc(files[index]); err != nil {
+ wrappedErr := wrapError(files[index], err)
+ if wrappedErr == nil {
+ // Keep failed deletions observable even if the wrapper suppresses the error.
Review Comment:
**minor** — Defensive branch for a condition no caller can produce, plus a test that exists only to cover it
`if wrappedErr == nil { wrappedErr = err }` guards against a wrapError that returns nil. Both production call sites (orphan_cleanup.go:731 and :1409) unconditionally return a non-nil fmt.Errorf, so this is unreachable in production; the only caller that can trigger it is TestDeleteFilesParallelPreservesErrorWithNilWrapper, written for that purpose. The prior review asked for the bench's wrapError to be fixed (it was); this production fallback is extra. AGENTS/house style: don't add error handling for scenarios that can't happen. Suggest dropping both the branch and the test.
##########
io/io.go:
##########
@@ -59,6 +59,7 @@ type IO interface {
// Remove removes the named file or (empty) directory.
//
// If there is an error, it will be of type *PathError.
+ // Implementations must be safe for concurrent use by multiple goroutines.
Remove(name string) error
Review Comment:
**minor** — Retroactive concurrency contract on the public IO interface deserves a release note
'Implementations must be safe for concurrent use by multiple goroutines.' tightens the contract of an exported extension point. Downstream users who registered a custom IO via io.Register and whose Remove was only ever driven serially by PurgeFiles will now be called from up to 32 goroutines. Every in-repo implementation already complies, so nothing breaks here, but this is a behavioural change for third-party implementors and should be called out in the changelog rather than only in a doc comment.
##########
table/orphan_cleanup_test.go:
##########
@@ -1616,6 +1623,256 @@ func (m mockFileInfo) ModTime() time.Time { return m.modTime }
func (m mockFileInfo) IsDir() bool { return m.mode.IsDir() }
func (m mockFileInfo) Sys() any { return nil }
+type purgeDeleteTrackingIO struct {
+ mockListableIO
+
+ targetActive int
+ reached chan struct{}
+ release chan struct{}
+
+ mu sync.Mutex
+ active int
+ maxActive int
+ removed []string
+ reachedOnce sync.Once
+ releaseOnce sync.Once
+}
+
+func (m *purgeDeleteTrackingIO) Remove(name string) error {
+ m.mu.Lock()
+ m.active++
+ if m.active > m.maxActive {
+ m.maxActive = m.active
+ }
+ if m.targetActive > 0 && m.active >= m.targetActive {
+ m.reachedOnce.Do(func() { close(m.reached) })
+ }
+ m.mu.Unlock()
+
+ if m.targetActive > 0 {
+ <-m.release
+ }
+
+ m.mu.Lock()
+ m.active--
+ m.removed = append(m.removed, name)
+ m.mu.Unlock()
+
+ return nil
+}
+
+func (m *purgeDeleteTrackingIO) MaxActive() int {
+ m.mu.Lock()
+ defer m.mu.Unlock()
+
+ return m.maxActive
+}
+
+func (m *purgeDeleteTrackingIO) Release() {
+ m.releaseOnce.Do(func() { close(m.release) })
+}
+
+func TestDeleteFilesParallelCollectsPurgeErrors(t *testing.T) {
+ const (
+ firstPath = "s3://bucket/table/first.parquet"
+ missingPath = "s3://bucket/table/missing.parquet"
+ slowPath = "s3://bucket/table/slow.parquet"
+ fastPath = "s3://bucket/table/fast.parquet"
+ )
+
+ slowErr := errors.New("slow removal failed")
+ fastErr := errors.New("fast removal failed")
+ files := []string{firstPath, missingPath, slowPath, fastPath}
+ var mu sync.Mutex
+ calls := make(map[string]int)
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 4,
+ func(path string) error {
+ mu.Lock()
+ calls[path]++
+ mu.Unlock()
+
+ var err error
+ switch path {
+ case missingPath:
+ err = stdfs.ErrNotExist
+ case slowPath:
+ time.Sleep(10 * time.Millisecond)
+
+ err = slowErr
+ case fastPath:
+ err = fastErr
+ }
+ if os.IsNotExist(err) {
+ return nil
+ }
+
+ return err
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ assert.Equal(t, []string{firstPath, missingPath}, deleted)
+ require.ErrorIs(t, err, slowErr)
+ require.ErrorIs(t, err, fastErr)
+ assert.NotContains(t, err.Error(), missingPath)
+ assert.Less(t, strings.Index(err.Error(), slowPath), strings.Index(err.Error(), fastPath))
+
+ mu.Lock()
+ assert.Equal(t, map[string]int{
+ firstPath: 1,
+ missingPath: 1,
+ slowPath: 1,
+ fastPath: 1,
+ }, calls)
+ mu.Unlock()
+}
+
+func TestDeleteFilesParallelPreservesErrorWithNilWrapper(t *testing.T) {
+ deleteErr := errors.New("delete failed")
+ files := []string{
+ "s3://bucket/table/ok.parquet",
+ "s3://bucket/table/failed.parquet",
+ }
+
+ deleted, err := deleteFilesParallel(
+ context.Background(),
+ files,
+ 2,
+ func(path string) error {
+ if path == files[1] {
+ return deleteErr
+ }
+
+ return nil
+ },
+ func(string, error) error { return nil },
+ )
+
+ assert.Equal(t, []string{files[0]}, deleted)
+ require.ErrorIs(t, err, deleteErr)
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnCancellation(t *testing.T) {
+ ctx, cancel := context.WithCancel(context.Background())
+ cancel()
+
+ called := false
+ _, err := deleteFilesParallel(
+ ctx,
+ []string{"s3://bucket/table/file.parquet"},
+ 4,
+ func(string) error {
+ called = true
+
+ return nil
+ },
+ func(path string, err error) error {
+ return fmt.Errorf("failed to remove %s: %w", path, err)
+ },
+ )
+
+ require.ErrorIs(t, err, context.Canceled)
+ assert.False(t, called)
+}
+
+func TestDeleteFilesParallelStopsQueuedWorkOnMidFlightCancellation(t *testing.T) {
+ const (
Review Comment:
**major** — Mid-flight cancellation test still does not reach the post-receive ctx guard
TestDeleteFilesParallelStopsQueuedWorkOnMidFlightCancellation calls cancel() and only then close(release), so ctx.Done() is always ready strictly before any worker can return to the `case index, ok := <-jobs` receive. The sender therefore always wins the select race and no worker ever reaches the inner guard at orphan_cleanup.go:801-806. This is the exact defect the prior review named ('would still pass if that guard were deleted') and it is still true. Fix: close(release) BEFORE cancel() (or release a subset first) so workers are contending on the jobs receive at the moment of cancellation, and assert deleteFunc is not invoked again after cancellation is observed.
<details><summary>Evidence</summary>
```text
Deleted the guard block at orphan_cleanup.go:801-806 -> `go test -count=200 -run TestDeleteFilesParallelStopsQueuedWorkOnMidFlightCancellation ./table/` => `ok ... 0.582s`; full `go test -race -count=1 ./table/` => `ok ... 20.047s`. Instrumented guard with an atomic counter: 'guard fired total: 0' across TestDeleteFiles*/TestPurgeFiles*/TestOrphan*. An unbarriered probe on the same code recorded 119 firings across 300 cancelled runs, proving the guard is reachable and load-bearing -- just never by the suite.
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): parallelize non-bulk PurgeFiles deletion [iceberg-go]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka merged PR #1973:
URL: https://github.com/apache/iceberg-go/pull/1973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org