You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "KranzL (via GitHub)" <gi...@apache.org> on 2026/09/22 04:21:29 UTC

[PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

KranzL opened a new pull request, #2041:
URL: https://github.com/apache/iceberg-go/pull/2041

   This PR depends on #2039 and carries its commits until that PR merges. It adds concurrent compaction groups on top of the bounded scan reorder heap.
   
   What changed
   
   RewriteDataFilesOptions has a MaxConcurrency field (table/rewrite_data_files.go:222-237). Zero and one keep the sequential loop. Values above one run ExecuteCompactionGroup calls under a bounded errgroup and apply results in the original group order, so manifests and RewriteResult counters match a sequential run. The atomic path branches at :372 and the partial-progress path branches per batch at :712; both call the shared helper at :451-478, which sets the errgroup limit to min(MaxConcurrency, len(groups)). A negative value returns ErrInvalidOperation at :358-359, the same way MaxCommits below zero does. The sizing bound is in the MaxConcurrency doc comment; website/src/api.md:501-515 shows RewriteDataFiles with empty options and lists no options, so that page is unchanged.
   
   Tests are TestRewriteDataFiles_MaxConcurrency* in table/rewrite_data_files_test.go:1566-1882: equal committed state and counters vs sequential, deterministic manifest order across runs, group failure returned with partial-progress outputs removed, context cancellation returned, in-flight groups capped at MaxConcurrency and exactly one for 0/1, negative rejected.
   
   The benchmark is BenchmarkRewriteDataFilesGroupConcurrency in table/rewrite_data_files_bench_test.go:184-243. Setup builds one partitioned table with 8 partitions of 8 files and 30000 rows each (1,920,000 rows, columns id int64, data string, payload 96-char hex string, score float64, about 223 MB). Each iteration rewrites all 8 groups on a fresh transaction and deletes its outputs afterwards, outside the timer.
   
   Measured evidence
   
   Command: go test ./table/ -run xxx -bench BenchmarkRewriteDataFilesGroupConcurrency -benchmem -count=5
   Hardware: Apple M3 Pro, 11 cores, go1.25.9 darwin/arm64
   
   | MaxConcurrency | ms/op, 5 runs | rows/s median | B/op median |
   | 1 | 1540, 1518, 1483, 1469, 1470 | 1.29M | 3.17 GB |
   | 2 | 778, 771, 772, 774, 773 | 2.49M | 3.15 GB |
   | 4 | 486, 460, 475, 468, 466 | 4.10M | 3.08 GB |
   | 8 | 399, 389, 383, 388, 385 | 4.95M | 3.06 GB |
   
   Median wall time falls from 1483 ms at 1 to 388 ms at 8, a 3.8x speedup. Branch MaxConcurrency=1 sits at 1483 ms against 1459 ms for the sequential-only run of the same fixture on the item 2 branch (2ff58a4, runs 1459, 1463, 1448, 1455, 1461), so the sequential path is unchanged within run noise. B/op is flat near 3.1 GB because every concurrency rewrites the same rows.
   
   I also ran a lagging-head check through RewriteDataFiles with 4 groups of 32 files, 4 scan workers per group, and each group's head task gated (scratch test, not committed). MaxConcurrency=1 held 3 of 3 expected tasks at 2.12 MB peak heap delta; MaxConcurrency=4 held 12 of 12 at 8.41 MB, which is 3.97 times the single-group value. The scan-level benchmark on this branch reports 3 gated batches and 2.12, 2.17, 2.38 MB for M 8, 32, 128, so the per-group bound from #2039 holds per group and multiplies by MaxConcurrency.
   
   Memory sizing
   
   Peak record-pipeline memory is about MaxConcurrency x (workers x (rows in the largest task + n) + (recordBatchBufferSize + 2) x n) rows, times the average row width in bytes. workers, n, and recordBatchBufferSize are the per-group GroupOptions values. Delete-side memory is outside this bound.
   
   Correctness
   
   The first group error cancels the groups still running through the errgroup context and is returned. The atomic path does not remove outputs on error, which matches its current behavior. The partial-progress path waits for every goroutine in the batch then passes each result, including a failed result carrying partial NewDataFiles, to cleanupBatch before the error returns.
   
   Validation
   
   go test ./table/ -race -count=1 -run 'RewriteDataFiles|ExecuteCompactionGroup|Rolling|Fanout|Clustered' passes in 9.8s. go test ./table/... -race -count=1 passes (table 30.9s, compaction 11.5s, dv 2.6s, internal 3.6s, substrait 4.1s). make lint reports 0 issues; go vet and gofmt are clean.
   
   Fixes #2040.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4158463257


##########
table/rewrite_data_files.go:
##########
@@ -348,31 +379,38 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
-			return result, err
-		}
-
-		if len(group.Tasks) == 0 {
-			continue
-		}
+	fs, err := t.tbl.fsF(ctx)

Review Comment:
   This doesn't fully close [r4108885244](https://github.com/apache/iceberg-go/pull/2041#discussion_r4108885244). The handle is opened with the caller's `ctx`, and the blob-backed IOs keep that ctx: `blobfs.New(ctx, …)` (reached via `io.LoadFSFunc`) stores it, and `Remove` calls `bfs.Delete(bfs.ctx, key)` (io/gocloud/blobfs/blob.go:309-315). Once the caller cancels, every `Remove` in `cleanupAtomicRewriteOutputs` fails with `context.Canceled`, so on S3/GCS/Azure the cancel path still orphans all outputs. I reproduced it with a test IO whose `Remove` honors the ctx passed to `fsF`: a sequential run cancelled after group 0 wrote goes from 4 parquet files before to 5 after, and the error ends with `clean up atomic rewrite outputs: remove …: context canceled`.
   
   Open the cleanup handle detached from cancellation. The partial path has the same pattern at :746 (outside this diff); please change it there too.
   
   ```suggestion
   	fs, err := t.tbl.fsF(context.WithoutCancel(ctx))
   ```
   
   `CleanupAfterCancelUsesOpenFS` can't catch this because `blockPathIO.Remove` is plain `LocalFS` and ignores ctx. Have `cancelOnDoneFSF` (rewrite_data_files_test.go:1806) wrap the IO so `Remove` fails once the ctx it was created with is done; then the test pins the real behavior.
   



##########
table/rewrite_data_files.go:
##########
@@ -401,12 +439,96 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	}
 
 	if err := rewrite.Commit(ctx); err != nil {
-		return result, fmt.Errorf("commit compaction: %w", err)
+		return result, cleanupAtomicRewriteOutputs(fs, applied, fmt.Errorf("commit compaction: %w", err))
 	}
 
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(fs iceio.IO, results []CompactionGroupResult, cause error) error {
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}
+	var g errgroup.Group
+	g.SetLimit(limit)
+	runCtx, cancelRuns := context.WithCancel(ctx)
+	defer cancelRuns()
+	results := make([]CompactionGroupResult, len(groups))
+	groupErrs := make([]error, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(runCtx, tbl, group, groupOpts...)
+			results[i] = gr
+			groupErrs[i] = err
+			if err != nil {
+				slog.Warn("compaction group failed", "index", i, "err", err)
+				cancelRuns()
+			}
+
+			return err
+		})
+	}
+	if err := g.Wait(); err != nil {
+		var firstErr, firstNonContextErr error
+		for _, groupErr := range groupErrs {
+			if groupErr == nil {
+				continue
+			}
+			if firstErr == nil {
+				firstErr = groupErr
+			}
+			if !errors.Is(groupErr, context.Canceled) && !errors.Is(groupErr, context.DeadlineExceeded) {

Review Comment:
   `runCtx` is a plain `WithCancel`, so cancelling the remaining groups only ever produces `context.Canceled`. A `DeadlineExceeded` here is either the caller's deadline, which the substitution at :522 already handles, or a real timeout inside that group (SDK and `http.Client` timeouts wrap it). Treating it as sibling cancellation hides the real failure: with group 2 failing on `open …: object store read timeout: context deadline exceeded` and the caller ctx still live, this returns `write compacted files for group "p0": context canceled`, while a sequential run returns group 2's timeout. That's what's left of [r4108885251](https://github.com/apache/iceberg-go/pull/2041#discussion_r4108885251), and it contradicts the "failures match a sequential run" doc at :226-230.
   
   ```suggestion
   			if !errors.Is(groupErr, context.Canceled) {
   ```
   



##########
table/rewrite_data_files.go:
##########
@@ -401,12 +439,96 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	}
 
 	if err := rewrite.Commit(ctx); err != nil {
-		return result, fmt.Errorf("commit compaction: %w", err)
+		return result, cleanupAtomicRewriteOutputs(fs, applied, fmt.Errorf("commit compaction: %w", err))
 	}
 
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(fs iceio.IO, results []CompactionGroupResult, cause error) error {
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}

Review Comment:
   main enabled the `modernize` linter in #2072 after this branch's last CI run. Merged with current main, modernize reports `rewrite_data_files.go:475:5: if statement can be modernized using max`, so the golangci-lint step fails once you rebase. #2072 rewrote the same pattern in table/internal/variant_shredding.go.
   
   ```suggestion
   	limit := max(min(maxConcurrentGroups, len(groups)), 1)
   ```
   



##########
table/rewrite_data_files.go:
##########
@@ -639,26 +761,26 @@ func (t *Transaction) rewriteDataFilesPartial(ctx context.Context, groups []Comp
 			return cause
 		}
 
-		for _, group := range batchGroups {
-			if err := ctx.Err(); err != nil {
-				return result, cleanupBatch(err)
-			}
-
-			gr, err := ExecuteCompactionGroup(ctx, current, group, opts.GroupOptions...)
+		if opts.MaxConcurrentGroups > 1 {
+			results, err := executeCompactionGroups(ctx, current, batchGroups, opts.GroupOptions, opts.MaxConcurrentGroups)

Review Comment:
   In partial-progress mode this only runs groups concurrently inside one commit batch, and batches still run one after another with a catalog commit in between. A batch is `ceil(groups/MaxCommits)` groups (:731), so with the default `MaxCommits` (10) and 10 or fewer groups every batch holds one group and `MaxConcurrentGroups` does nothing. With 8 groups, `PartialProgress`, `MaxConcurrentGroups: 4` and scan concurrency 1, peak concurrent data-file opens is 1 at `MaxCommits: 0` and 3-4 at `MaxCommits: 1`, which is why the partial cases in the tests set `MaxCommits: 1`. Java's `max-concurrent-file-group-rewrites` doesn't depend on commit batching. At minimum, state this cap in the `MaxConcurrentGroups` doc (:222-244); overlapping execution across batches can be a follow-up.
   



##########
table/rewrite_data_files.go:
##########
@@ -401,12 +439,96 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	}
 
 	if err := rewrite.Commit(ctx); err != nil {
-		return result, fmt.Errorf("commit compaction: %w", err)
+		return result, cleanupAtomicRewriteOutputs(fs, applied, fmt.Errorf("commit compaction: %w", err))
 	}
 
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(fs iceio.IO, results []CompactionGroupResult, cause error) error {
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}
+	var g errgroup.Group
+	g.SetLimit(limit)
+	runCtx, cancelRuns := context.WithCancel(ctx)
+	defer cancelRuns()
+	results := make([]CompactionGroupResult, len(groups))
+	groupErrs := make([]error, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(runCtx, tbl, group, groupOpts...)
+			results[i] = gr
+			groupErrs[i] = err
+			if err != nil {
+				slog.Warn("compaction group failed", "index", i, "err", err)
+				cancelRuns()

Review Comment:
   Nit: every group that gets cancelled because a sibling failed also logs here, so one real failure with N groups in flight produces N `compaction group failed` warnings, N-1 of them `context canceled`. Skip the log (still calling `cancelRuns()`) when `errors.Is(err, context.Canceled) && runCtx.Err() != nil`, or log those at Debug.
   



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1426,749 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	return newMaxConcPartitionedTableWithFSF(t, func(context.Context) (iceio.IO, error) { return fs, nil })
+}
+
+func newMaxConcPartitionedTableWithFSF(t *testing.T, fsF func(context.Context) (iceio.IO, error)) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		fsF,
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func idsByPartitionValue(t *testing.T, tbl *table.Table) map[string][]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string][]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		dataIdx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, dataIdx)
+		idIdx := rec.Schema().FieldIndices("id")
+		require.NotEmpty(t, idIdx)
+		dataCol, ok := rec.Column(dataIdx[0]).(*array.String)
+		require.True(t, ok)
+		idCol, ok := rec.Column(idIdx[0]).(*array.Int64)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[dataCol.Value(i)] = append(out[dataCol.Value(i)], idCol.Value(i))
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrentGroupsMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrentGroups: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	idsSeq := idsByPartitionValue(t, committedSeq)
+	idsConc := idsByPartitionValue(t, committedConc)
+	require.Len(t, idsConc, 8)
+	for p := range 8 {
+		key := fmt.Sprintf("p%d", p)
+		assert.ElementsMatch(t, idsSeq[key], idsConc[key])
+		assert.Len(t, idsConc[key], 10)
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrentGroupsNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrentGroups: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrentGroups: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrentGroupsDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrentGroups: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrentGroups: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {

Review Comment:
   Nit, non-blocking: `failOpenIO`, `blockPathIO`, `orderFailIO` and `gateOpenIO` all intercept `Open` by path substring; one IO with an `onOpen func(name string) error` hook would replace the four. `newMaxConcPartitionedTableWithFSF` duplicates `newPartialProgressPartitionedTable` (:871) except for the identifier and fsF, `allParquetFiles` can replace `parquetFiles`, `manifestDataPartitions` and `manifestLiveDataPaths` are the same loop, and the bench reimplements `groupsByPartition`. Folding these together should cut 150+ lines without losing coverage.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4087862500


##########
table/rewrite_data_files.go:
##########
@@ -332,31 +369,51 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
+	if opts.MaxConcurrency > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrency)
+		if err != nil {

Review Comment:
   Done in 30ffe36. The concurrent branch passes its results to cleanupAtomicRewriteOutputs at table/rewrite_data_files.go:382, and the sequential branch tracks applied results and cleans them at :391 and :400. The helper at :453 opens the IO through t.tbl.fsF and joins a cleanup failure with the cause instead of replacing it.



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())

Review Comment:
   Done in ad900a5. The atomic half captures beforeAtomicFiles and asserts ElementsMatch after the error at table/rewrite_data_files_test.go:1722, the same check the partial half uses. go test ./table/ -race -count=10 -run 'TestRewriteDataFiles_MaxConcurrentGroups(GroupFailure|LimitsInFlight|ContextCancel)' passes in 11.2s.



##########
table/rewrite_data_files.go:
##########
@@ -391,6 +448,35 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrency int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(min(maxConcurrency, len(groups)))
+	results := make([]CompactionGroupResult, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(gctx, tbl, group, groupOpts...)
+			results[i] = gr
+
+			return err
+		})
+	}
+	if err := g.Wait(); err != nil {
+		if ctx.Err() != nil {

Review Comment:
   Done in 30ffe36, first option. The caller context error replaces the group error only when the group error is itself context-caused, at table/rewrite_data_files.go:488. A data failure that races a cancellation keeps its real error now. The cancel test still asserts exact equality at table/rewrite_data_files_test.go:1828.



##########
table/rewrite_data_files.go:
##########
@@ -391,6 +448,35 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrency int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(min(maxConcurrency, len(groups)))

Review Comment:
   Done in 30ffe36 at table/rewrite_data_files.go:469-472, the clamp as suggested.



##########
table/arrow_scanner.go:
##########
@@ -1242,9 +1242,61 @@ func (as *arrowScan) addTaskProjectedFieldIDs(invariants *arrowScanInvariants, t
 }
 
 type enumeratedRecord struct {
-	Record tblutils.Enumerated[arrow.RecordBatch]
-	Task   tblutils.Enumerated[FileScanTask]
-	Err    error
+	Record  tblutils.Enumerated[arrow.RecordBatch]
+	Task    tblutils.Enumerated[FileScanTask]
+	Err     error
+	credits taskCredits
+}
+
+const maxInFlightTasksPerWorker = 1
+
+type taskCredits chan struct{}
+
+func newTaskCredits() taskCredits {
+	return make(taskCredits, maxInFlightTasksPerWorker)
+}
+
+func (c taskCredits) acquire(ctx context.Context) error {
+	if c == nil {
+		return nil
+	}
+
+	select {
+	case c <- struct{}{}:
+		return nil
+	case <-ctx.Done():
+		return context.Cause(ctx)
+	}
+}
+
+func (c taskCredits) release() {
+	if c != nil {
+		<-c
+	}
+}
+
+type recordSink struct {
+	out     chan<- enumeratedRecord
+	credits taskCredits
+}
+
+func newRecordSink(out chan<- enumeratedRecord) recordSink {
+	return recordSink{out: out, credits: newTaskCredits()}
+}
+
+func (s recordSink) reserve(ctx context.Context) error {
+	return s.credits.acquire(ctx)
+}
+
+func (s recordSink) send(rec enumeratedRecord) {
+	if rec.Record.Last {

Review Comment:
   This is #2039's code. The same point was raised there and reworked in 149c395: reserve marks the credit held and the sink hands it to exactly one record, the Last record in send or the error record in fail when no Last record was sent, and TestRecordSinkHandsOffCreditOnce covers fail without Last, Last followed by fail, and reserve blocking while the credit is out. The thread on #2039 has the details. This hunk drops out of the diff when #2039 merges and this rebases.



##########
table/rewrite_data_files.go:
##########
@@ -332,31 +369,51 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
+	if opts.MaxConcurrency > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrency)
+		if err != nil {
 			return result, err
 		}
-
-		if len(group.Tasks) == 0 {
-			continue
+		for _, gr := range results {

Review Comment:
   Done in 30ffe36. Both atomic branches call applyAtomicGroupResult at table/rewrite_data_files.go:439, and both partial branches call appendPartialGroupResult at :843.



##########
table/rewrite_data_files.go:
##########
@@ -217,6 +218,23 @@ type RewriteDataFilesOptions struct {
 	// size, scan concurrency). See the With* helpers returning
 	// [CompactionGroupOption].
 	GroupOptions []CompactionGroupOption
+
+	// MaxConcurrency bounds how many compaction groups run at once.
+	// Zero and one both mean sequential execution, which is the default.
+	// Larger values run [ExecuteCompactionGroup] calls under a bounded
+	// errgroup and apply their results in the original group order, so
+	// manifests and [RewriteResult] are identical to a sequential run.
+	// The first error cancels the groups still running and is returned.
+	// Peak record-pipeline memory is MaxConcurrency times the per-group
+	// bound stated on [WithCompactionArrowBatchSize]:
+	//
+	//	MaxConcurrency x (workers x (rows in the largest task + n) + (recordBatchBufferSize + 2) x n)
+	//
+	// rows, where workers, n and recordBatchBufferSize are the per-group
+	// values. Multiply rows by the average row width in bytes for a byte
+	// estimate. Delete-side memory is outside this bound. Negative values
+	// are rejected with [ErrInvalidOperation].
+	MaxConcurrency int

Review Comment:
   Renamed in 50480ae. The field is MaxConcurrentGroups at table/rewrite_data_files.go:241, both docs cross-reference each other, and both state the N times scan-workers file-open fan-out (:235-239 and :277-280). Issue #2040 and the PR body use the new name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4087970092


##########
table/arrow_scanner.go:
##########
@@ -2190,6 +2244,7 @@ func createIteratorWithCleanup(ctx context.Context, numWorkers uint, records <-c
 					return
 				}
 
+				enum.credits.release()

Review Comment:
   This line is #2039's code, unchanged in this PR so the rebase stays clean. The release-before-yield point was not raised on #2039 itself; I will settle it there before #2039 merges. This hunk drops out of the diff on rebase.



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))

Review Comment:
   Done in ad900a5. idsByPartitionValue at table/rewrite_data_files_test.go:1505 collects the id values per data partition, and the test compares per-partition sets with ElementsMatch plus a count of 10 at :1623-1630.



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())
+
+	fsPartial := &failOpenIO{}
+	tblPartial := newMaxConcPartitionedTable(t, fsPartial)
+	tblPartial = addMaxConcPartitions(t, tblPartial, 4, 1, 5)
+	groupsPartial := groupsByPartition(t, tblPartial)
+	require.Len(t, groupsPartial, 4)
+	fsPartial.setFail(groupsPartial[1].Tasks[0].File.FilePath(), injected)
+	beforeFiles := allParquetFiles(t, tblPartial.Location())
+
+	txPartial := tblPartial.NewTransaction()
+	result, err := txPartial.RewriteDataFiles(t.Context(), groupsPartial, table.RewriteDataFilesOptions{
+		PartialProgress: true,
+		MaxCommits:      1,
+		MaxConcurrency:  4,
+	})
+	require.Error(t, err)
+	require.NotNil(t, result)
+	assert.Contains(t, err.Error(), injected.Error())
+	assert.Empty(t, result.CompletedGroups)
+	assert.ElementsMatch(t, beforeFiles, allParquetFiles(t, tblPartial.Location()))
+}
+
+type gateOpenIO struct {
+	iceio.LocalFS
+	mu      sync.Mutex
+	enabled bool
+	entered chan struct{}
+	release chan struct{}
+}
+
+func (g *gateOpenIO) enable() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	g.enabled = true
+	g.entered = make(chan struct{}, 32)
+	g.release = make(chan struct{})
+}
+
+func (g *gateOpenIO) releaseAll() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	select {
+	case <-g.release:
+	default:
+		close(g.release)
+	}
+}
+
+func (g *gateOpenIO) Open(name string) (iceio.File, error) {
+	g.mu.Lock()
+	enabled, entered, release := g.enabled, g.entered, g.release
+	g.mu.Unlock()
+	if enabled && strings.Contains(name, "/data/") {
+		select {
+		case <-release:
+		default:
+			select {
+			case entered <- struct{}{}:
+			default:
+			}
+			<-release
+		}
+	}
+
+	return g.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyContextCancel(t *testing.T) {
+	for _, partial := range []bool{false, true} {
+		gate := &gateOpenIO{}
+		tbl := newMaxConcPartitionedTable(t, gate)
+		tbl = addMaxConcPartitions(t, tbl, 8, 1, 10)
+		groups := groupsByPartition(t, tbl)
+		require.Len(t, groups, 8)
+		gate.enable()
+
+		ctx, cancel := context.WithCancel(t.Context())
+		done := make(chan struct{})
+		var rewriteErr error
+		go func() {
+			defer close(done)
+			tx := tbl.NewTransaction()
+			opts := table.RewriteDataFilesOptions{MaxConcurrency: 4}
+			if partial {
+				opts.PartialProgress = true
+				opts.MaxCommits = 1
+			}
+			_, rewriteErr = tx.RewriteDataFiles(ctx, groups, opts)
+		}()
+
+		for range 4 {
+			select {
+			case <-gate.entered:
+			case <-done:
+				t.Fatalf("rewrite finished before 4 groups were in flight, err=%v", rewriteErr)
+			case <-t.Context().Done():
+				t.Fatal("test context done while waiting for groups")
+			}
+		}
+		cancel()
+		gate.releaseAll()
+		<-done
+		require.Error(t, rewriteErr)
+		assert.ErrorIs(t, rewriteErr, context.Canceled)
+		assert.Equal(t, ctx.Err(), rewriteErr)
+	}
+}
+
+type countOpenFile struct {
+	iceio.File
+	owner *countOpenIO
+	once  sync.Once
+}
+
+func (f *countOpenFile) Close() error {
+	err := f.File.Close()
+	f.once.Do(func() {
+		f.owner.mu.Lock()
+		defer f.owner.mu.Unlock()
+		f.owner.cur--
+	})
+
+	return err
+}
+
+type countOpenIO struct {
+	iceio.LocalFS
+	mu   sync.Mutex
+	cur  int
+	peak int
+}
+
+func (c *countOpenIO) Open(name string) (iceio.File, error) {
+	f, err := c.LocalFS.Open(name)
+	if err != nil {
+		return nil, err
+	}
+	if strings.Contains(name, "/data/") {
+		c.mu.Lock()
+		c.cur++
+		if c.cur > c.peak {
+			c.peak = c.cur
+		}
+		c.mu.Unlock()
+		time.Sleep(20 * time.Millisecond)

Review Comment:
   Done in ad900a5. countOpenIO.Open blocks at table/rewrite_data_files_test.go:1880 until two /data/ opens overlap, and the maxConc=4 case enables the barrier at :1922. The sequential cases leave it off and still assert peak 1. No wall clock involved; -race -count=10 on the three concurrency tests passes in 11.2s.



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())
+
+	fsPartial := &failOpenIO{}
+	tblPartial := newMaxConcPartitionedTable(t, fsPartial)
+	tblPartial = addMaxConcPartitions(t, tblPartial, 4, 1, 5)
+	groupsPartial := groupsByPartition(t, tblPartial)
+	require.Len(t, groupsPartial, 4)
+	fsPartial.setFail(groupsPartial[1].Tasks[0].File.FilePath(), injected)
+	beforeFiles := allParquetFiles(t, tblPartial.Location())
+
+	txPartial := tblPartial.NewTransaction()
+	result, err := txPartial.RewriteDataFiles(t.Context(), groupsPartial, table.RewriteDataFilesOptions{
+		PartialProgress: true,
+		MaxCommits:      1,
+		MaxConcurrency:  4,
+	})
+	require.Error(t, err)
+	require.NotNil(t, result)
+	assert.Contains(t, err.Error(), injected.Error())
+	assert.Empty(t, result.CompletedGroups)
+	assert.ElementsMatch(t, beforeFiles, allParquetFiles(t, tblPartial.Location()))
+}
+
+type gateOpenIO struct {
+	iceio.LocalFS
+	mu      sync.Mutex
+	enabled bool
+	entered chan struct{}
+	release chan struct{}
+}
+
+func (g *gateOpenIO) enable() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	g.enabled = true
+	g.entered = make(chan struct{}, 32)
+	g.release = make(chan struct{})
+}
+
+func (g *gateOpenIO) releaseAll() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	select {
+	case <-g.release:
+	default:
+		close(g.release)
+	}
+}
+
+func (g *gateOpenIO) Open(name string) (iceio.File, error) {
+	g.mu.Lock()
+	enabled, entered, release := g.enabled, g.entered, g.release
+	g.mu.Unlock()
+	if enabled && strings.Contains(name, "/data/") {
+		select {
+		case <-release:
+		default:
+			select {
+			case entered <- struct{}{}:
+			default:
+			}
+			<-release
+		}
+	}
+
+	return g.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyContextCancel(t *testing.T) {
+	for _, partial := range []bool{false, true} {
+		gate := &gateOpenIO{}
+		tbl := newMaxConcPartitionedTable(t, gate)
+		tbl = addMaxConcPartitions(t, tbl, 8, 1, 10)
+		groups := groupsByPartition(t, tbl)
+		require.Len(t, groups, 8)
+		gate.enable()
+
+		ctx, cancel := context.WithCancel(t.Context())
+		done := make(chan struct{})
+		var rewriteErr error
+		go func() {
+			defer close(done)
+			tx := tbl.NewTransaction()
+			opts := table.RewriteDataFilesOptions{MaxConcurrency: 4}
+			if partial {
+				opts.PartialProgress = true
+				opts.MaxCommits = 1
+			}
+			_, rewriteErr = tx.RewriteDataFiles(ctx, groups, opts)
+		}()
+
+		for range 4 {
+			select {
+			case <-gate.entered:
+			case <-done:
+				t.Fatalf("rewrite finished before 4 groups were in flight, err=%v", rewriteErr)
+			case <-t.Context().Done():
+				t.Fatal("test context done while waiting for groups")
+			}
+		}
+		cancel()
+		gate.releaseAll()
+		<-done
+		require.Error(t, rewriteErr)
+		assert.ErrorIs(t, rewriteErr, context.Canceled)
+		assert.Equal(t, ctx.Err(), rewriteErr)
+	}
+}
+
+type countOpenFile struct {
+	iceio.File
+	owner *countOpenIO
+	once  sync.Once
+}
+
+func (f *countOpenFile) Close() error {
+	err := f.File.Close()
+	f.once.Do(func() {
+		f.owner.mu.Lock()
+		defer f.owner.mu.Unlock()
+		f.owner.cur--
+	})
+
+	return err
+}
+
+type countOpenIO struct {
+	iceio.LocalFS
+	mu   sync.Mutex
+	cur  int
+	peak int
+}
+
+func (c *countOpenIO) Open(name string) (iceio.File, error) {
+	f, err := c.LocalFS.Open(name)
+	if err != nil {
+		return nil, err
+	}
+	if strings.Contains(name, "/data/") {
+		c.mu.Lock()
+		c.cur++
+		if c.cur > c.peak {
+			c.peak = c.cur
+		}
+		c.mu.Unlock()
+		time.Sleep(20 * time.Millisecond)
+
+		return &countOpenFile{File: f, owner: c}, nil
+	}
+
+	return f, nil
+}
+
+func (c *countOpenIO) reset() {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.cur = 0
+	c.peak = 0
+}
+
+func (c *countOpenIO) getPeak() int {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	return c.peak
+}
+
+func TestRewriteDataFiles_MaxConcurrencyLimitsInFlight(t *testing.T) {
+	for _, maxConc := range []int{4, 0, 1} {

Review Comment:
   Done in ad900a5: t.Run with maxConc=%d at table/rewrite_data_files_test.go:1915 and partial=%v at :1792.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4139890723


##########
table/rewrite_data_files.go:
##########
@@ -348,31 +376,32 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
-			return result, err
+	if opts.MaxConcurrentGroups > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrentGroups)
+		if err != nil {
+			return result, cleanupAtomicRewriteOutputs(ctx, t.tbl, results, err)

Review Comment:
   Done in 6c9a36e. The atomic path tracks applied results across both branches and the rewrite.Commit failure branch passes them to cleanupAtomicRewriteOutputs at table/rewrite_data_files.go:442. Unconditional cleanup is safe here since rewrite.Commit only stages onto the caller transaction while the catalog commit happens later in the caller Transaction.Commit. CommitFailureCleansOutputs in 1116187 fails the staging with an unsupported content type and asserts the before/after parquet set matches for maxConc 4 and 0.



##########
table/rewrite_data_files.go:
##########
@@ -407,6 +436,65 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(ctx context.Context, tbl *Table, results []CompactionGroupResult, cause error) error {
+	fs, err := tbl.fsF(ctx)

Review Comment:
   Done in 6c9a36e, second option. The atomic path opens fs once up front at table/rewrite_data_files.go:382 and cleanupAtomicRewriteOutputs takes the open iceio.IO handle instead of (ctx, tbl), mirroring the partial path. CleanupAfterCancelUsesOpenFS in 1116187 runs an fsF that fails on a done context and asserts full cleanup after mid-run cancellation on both the sequential and concurrent paths.



##########
table/rewrite_data_files.go:
##########
@@ -407,6 +436,65 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(ctx context.Context, tbl *Table, results []CompactionGroupResult, cause error) error {
+	fs, err := tbl.fsF(ctx)
+	if err != nil {
+		return errors.Join(cause, fmt.Errorf("open table IO to clean up atomic rewrite outputs: %w", err))
+	}
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(limit)
+	results := make([]CompactionGroupResult, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(gctx, tbl, group, groupOpts...)
+			results[i] = gr
+
+			return err
+		})
+	}
+	if err := g.Wait(); err != nil {
+		if ctxErr := ctx.Err(); ctxErr != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)) {

Review Comment:
   Done in 6c9a36e, lowest-index option plus logging. executeCompactionGroups records per-group errors, logs every non-nil one via slog.Warn, and returns the lowest-index failure not caused by sibling cancellation (falling back to the lowest-index failure when every group was canceled), so failures match a sequential run; the round-1 context substitution now applies to the selected error. Cancellation also broadcasts over a plain WithCancel context instead of errgroup.WithContext: with the shared cause, a still-running sibling scan iterator surfaced the failing group error value through context.Cause instead of context.Canceled, which the new ordering test caught before the broadcast change. FailureReturnsLowestIndexError in 1116187 orders two injected failures so index 3 finishes first and asserts index 1 error is returned while both appear in the log.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#issuecomment-5804068494

   All threads above are answered. 30ffe36 cleans up atomic outputs on failure, shares the per-group apply blocks through two helpers, narrows the context-error substitution and clamps the errgroup limit. 50480ae renames the option to MaxConcurrentGroups with cross-referenced docs. ad900a5 hardens the tests. #2039 is approved but not merged, so this still carries its commits; I will rebase onto main and drop the dependency sentence once it lands, which also removes the two arrow_scanner threads from this diff. BenchmarkRewriteDataFilesGroupConcurrency -count=5 passes after the rename; the evidence table is unchanged because the success path is untouched and the rerun box was loaded (load 25 on 11 cores). go test ./table/... -race -count=1 passes (table 41.8s, compaction 21.0s, dv 3.7s, internal 3.3s, substrait 2.7s); make lint reports 0 issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#issuecomment-5804817487

   Rebased onto origin/main at 5d377f1 now that #2039 merged as dce9757. The branch holds the 7 item 1 commits only (8d750e9..7eed7ee); the diff touches table/rewrite_data_files.go, table/rewrite_data_files_test.go and table/rewrite_data_files_bench_test.go, and the two arrow_scanner threads are out of the diff. The rebase applied with no conflicts and every cited line number is unchanged. go test ./table/... -race -count=1 passes (table 38.8s, compaction 14.9s, dv 5.8s, internal 5.8s, substrait 3.6s); BenchmarkRewriteDataFilesGroupConcurrency -count=5 passes with the 1 to 4 speedup visible even under load; make lint reports 0 issues.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4088389030


##########
table/arrow_scanner.go:
##########
@@ -2190,6 +2244,7 @@ func createIteratorWithCleanup(ctx context.Context, numWorkers uint, records <-c
 					return
 				}
 
+				enum.credits.release()

Review Comment:
   Correction to my reply above: #2039 merged as dce9757 while I was working through these threads, so this is a follow-up against main now instead of a pre-merge change on #2039.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "dustinCodes84600 (via GitHub)" <gi...@apache.org>.
dustinCodes84600 commented on PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#issuecomment-5821818986

   `cleanupAtomicRewriteOutputs` gets the already-canceled caller context, so an FS factory that honors cancellation can fail before deleting outputs. The cancellation test only checks returned error; worth also asserting files are removed and probably doing cleanup with `context.WithoutCancel` plus its own timeout.
   
   to keep the cancellation paths straight, I made a visual walkthrough while reading this PR: https://flyovers.dev/flyover/ddd0b368-9cd0-4a0d-831c-03d0867c9ed5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4139890862


##########
table/rewrite_data_files.go:
##########
@@ -407,6 +436,65 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(ctx context.Context, tbl *Table, results []CompactionGroupResult, cause error) error {
+	fs, err := tbl.fsF(ctx)

Review Comment:
   Done in 6c9a36e. The atomic path opens fs once up front at table/rewrite_data_files.go:382 and cleanupAtomicRewriteOutputs takes the open iceio.IO handle instead of (ctx, tbl), mirroring the partial path. CleanupAfterCancelUsesOpenFS in 1116187 runs an fsF that fails on a done context and asserts full cleanup after mid-run cancellation on both the sequential and concurrent paths.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4075722925


##########
table/rewrite_data_files.go:
##########
@@ -332,31 +369,51 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
+	if opts.MaxConcurrency > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrency)
+		if err != nil {

Review Comment:
   The concurrent atomic path leaks output files on failure, and the concurrency makes it strictly worse than the sequential loop.
   
   When a group fails here we return without touching `results`, so every group that already finished writing its compacted parquet is orphaned on disk (nothing gets committed to a manifest). Sequentially that's bounded to the groups before the failing one; with `MaxConcurrency`, up to N groups race ahead and finish real writes before cancellation propagates, so the leak scales with the knob.
   
   The partial-progress branch already handles this via `cleanupBatch(err, results...)`. I'd mirror it here: on error call `cleanupCompactionOutputs(fs, results)` (fs from `t.tbl.fsF(ctx)`) before returning, and do the same in the sequential `else` branch. wdyt?



##########
table/rewrite_data_files.go:
##########
@@ -391,6 +448,35 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrency int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(min(maxConcurrency, len(groups)))
+	results := make([]CompactionGroupResult, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(gctx, tbl, group, groupOpts...)
+			results[i] = gr
+
+			return err
+		})
+	}
+	if err := g.Wait(); err != nil {
+		if ctx.Err() != nil {

Review Comment:
   This masks the real failure whenever the caller's ctx happens to be done.
   
   If a group fails for an unrelated reason (bad data file, disk full) while the caller's ctx is also canceled, we discard the group's actual error and return a bare context error, so an operator debugging a production failure sees "context canceled" instead of the real cause.
   
   I'd only substitute `ctx.Err()` when the group's own error is context-caused (`errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)`), or just return `errors.Join(err, ctx.Err())` to keep both. The cancel test can switch to `assert.ErrorIs` instead of asserting equality. wdyt?



##########
table/rewrite_data_files.go:
##########
@@ -391,6 +448,35 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrency int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(min(maxConcurrency, len(groups)))

Review Comment:
   This relies entirely on both call sites gating with `MaxConcurrency > 1`.
   
   `errgroup.SetLimit` treats 0 as "no goroutine may run" (every `g.Go` blocks forever) and a negative as unbounded, so if a future caller or a refactor drops that guard we get either a deadlock or a silently unbounded fan-out, and neither is caught by the type system.
   
   Since the function is unexported and cheap to harden, I'd clamp defensively: `limit := min(maxConcurrency, len(groups)); if limit < 1 { limit = 1 }`. Thoughts?



##########
table/rewrite_data_files.go:
##########
@@ -332,31 +369,51 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
+	if opts.MaxConcurrency > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrency)
+		if err != nil {
 			return result, err
 		}
-
-		if len(group.Tasks) == 0 {
-			continue
+		for _, gr := range results {

Review Comment:
   This per-group apply block (skip empty, `ApplyResult`, `accumulateGroupMetrics`, stage `SafePosDeletes`/`SafeDeletionVectors`) is now copy-pasted four times: atomic concurrent and sequential here, plus the two partial-progress branches around :700. A future tweak to the accounting has to land in all four or the concurrent and sequential paths silently diverge, which is the exact thing `MaxConcurrency`'s doc promises won't happen.
   
   Could we factor the per-group apply into a small helper shared by both branches?
   
   ```go
   func applyGroupResult(rewrite *Rewrite, result *RewriteResult, staged map[string]struct{}, gr CompactionGroupResult) { ... }
   ```
   
   Same shape for the partial accumulation. Thoughts?



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())
+
+	fsPartial := &failOpenIO{}
+	tblPartial := newMaxConcPartitionedTable(t, fsPartial)
+	tblPartial = addMaxConcPartitions(t, tblPartial, 4, 1, 5)
+	groupsPartial := groupsByPartition(t, tblPartial)
+	require.Len(t, groupsPartial, 4)
+	fsPartial.setFail(groupsPartial[1].Tasks[0].File.FilePath(), injected)
+	beforeFiles := allParquetFiles(t, tblPartial.Location())
+
+	txPartial := tblPartial.NewTransaction()
+	result, err := txPartial.RewriteDataFiles(t.Context(), groupsPartial, table.RewriteDataFilesOptions{
+		PartialProgress: true,
+		MaxCommits:      1,
+		MaxConcurrency:  4,
+	})
+	require.Error(t, err)
+	require.NotNil(t, result)
+	assert.Contains(t, err.Error(), injected.Error())
+	assert.Empty(t, result.CompletedGroups)
+	assert.ElementsMatch(t, beforeFiles, allParquetFiles(t, tblPartial.Location()))
+}
+
+type gateOpenIO struct {
+	iceio.LocalFS
+	mu      sync.Mutex
+	enabled bool
+	entered chan struct{}
+	release chan struct{}
+}
+
+func (g *gateOpenIO) enable() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	g.enabled = true
+	g.entered = make(chan struct{}, 32)
+	g.release = make(chan struct{})
+}
+
+func (g *gateOpenIO) releaseAll() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	select {
+	case <-g.release:
+	default:
+		close(g.release)
+	}
+}
+
+func (g *gateOpenIO) Open(name string) (iceio.File, error) {
+	g.mu.Lock()
+	enabled, entered, release := g.enabled, g.entered, g.release
+	g.mu.Unlock()
+	if enabled && strings.Contains(name, "/data/") {
+		select {
+		case <-release:
+		default:
+			select {
+			case entered <- struct{}{}:
+			default:
+			}
+			<-release
+		}
+	}
+
+	return g.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyContextCancel(t *testing.T) {
+	for _, partial := range []bool{false, true} {
+		gate := &gateOpenIO{}
+		tbl := newMaxConcPartitionedTable(t, gate)
+		tbl = addMaxConcPartitions(t, tbl, 8, 1, 10)
+		groups := groupsByPartition(t, tbl)
+		require.Len(t, groups, 8)
+		gate.enable()
+
+		ctx, cancel := context.WithCancel(t.Context())
+		done := make(chan struct{})
+		var rewriteErr error
+		go func() {
+			defer close(done)
+			tx := tbl.NewTransaction()
+			opts := table.RewriteDataFilesOptions{MaxConcurrency: 4}
+			if partial {
+				opts.PartialProgress = true
+				opts.MaxCommits = 1
+			}
+			_, rewriteErr = tx.RewriteDataFiles(ctx, groups, opts)
+		}()
+
+		for range 4 {
+			select {
+			case <-gate.entered:
+			case <-done:
+				t.Fatalf("rewrite finished before 4 groups were in flight, err=%v", rewriteErr)
+			case <-t.Context().Done():
+				t.Fatal("test context done while waiting for groups")
+			}
+		}
+		cancel()
+		gate.releaseAll()
+		<-done
+		require.Error(t, rewriteErr)
+		assert.ErrorIs(t, rewriteErr, context.Canceled)
+		assert.Equal(t, ctx.Err(), rewriteErr)
+	}
+}
+
+type countOpenFile struct {
+	iceio.File
+	owner *countOpenIO
+	once  sync.Once
+}
+
+func (f *countOpenFile) Close() error {
+	err := f.File.Close()
+	f.once.Do(func() {
+		f.owner.mu.Lock()
+		defer f.owner.mu.Unlock()
+		f.owner.cur--
+	})
+
+	return err
+}
+
+type countOpenIO struct {
+	iceio.LocalFS
+	mu   sync.Mutex
+	cur  int
+	peak int
+}
+
+func (c *countOpenIO) Open(name string) (iceio.File, error) {
+	f, err := c.LocalFS.Open(name)
+	if err != nil {
+		return nil, err
+	}
+	if strings.Contains(name, "/data/") {
+		c.mu.Lock()
+		c.cur++
+		if c.cur > c.peak {
+			c.peak = c.cur
+		}
+		c.mu.Unlock()
+		time.Sleep(20 * time.Millisecond)

Review Comment:
   This peak-concurrency assertion leans on a fixed 20ms sleep to widen the race window, which is a real wall-clock race: under `-race` or a loaded CI runner it's plausible for fewer than 2 groups to overlap in that window and `assert.GreaterOrEqual(peak, 2)` flakes.
   
   The adjacent cancel test already uses a proper channel gate (`gateOpenIO`), and the scanner reorder tests use `testing/synctest`. I'd reuse one of those here for a deterministic sync point instead of the sleep. Thoughts?



##########
table/rewrite_data_files.go:
##########
@@ -217,6 +218,23 @@ type RewriteDataFilesOptions struct {
 	// size, scan concurrency). See the With* helpers returning
 	// [CompactionGroupOption].
 	GroupOptions []CompactionGroupOption
+
+	// MaxConcurrency bounds how many compaction groups run at once.
+	// Zero and one both mean sequential execution, which is the default.
+	// Larger values run [ExecuteCompactionGroup] calls under a bounded
+	// errgroup and apply their results in the original group order, so
+	// manifests and [RewriteResult] are identical to a sequential run.
+	// The first error cancels the groups still running and is returned.
+	// Peak record-pipeline memory is MaxConcurrency times the per-group
+	// bound stated on [WithCompactionArrowBatchSize]:
+	//
+	//	MaxConcurrency x (workers x (rows in the largest task + n) + (recordBatchBufferSize + 2) x n)
+	//
+	// rows, where workers, n and recordBatchBufferSize are the per-group
+	// values. Multiply rows by the average row width in bytes for a byte
+	// estimate. Delete-side memory is outside this bound. Negative values
+	// are rejected with [ErrInvalidOperation].
+	MaxConcurrency int

Review Comment:
   One foot-gun worth heading off: this `MaxConcurrency` (groups in parallel) sits right next to `WithCompactionScanConcurrency`, which forwards to `Table.Scan` as `WithMaxConcurrency` (scan workers per group). Two different knobs both called "max concurrency", and they compose multiplicatively, so `MaxConcurrency=N` with the scan default gives roughly N x GOMAXPROCS concurrent file opens, an easy connection/FD spike on object stores.
   
   The doc here only quantifies memory. I'd at minimum cross-reference the two in each other's doc and note the I/O fan-out; renaming to something like `MaxConcurrentGroups` would remove the ambiguity entirely if we're open to it. wdyt?



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())
+
+	fsPartial := &failOpenIO{}
+	tblPartial := newMaxConcPartitionedTable(t, fsPartial)
+	tblPartial = addMaxConcPartitions(t, tblPartial, 4, 1, 5)
+	groupsPartial := groupsByPartition(t, tblPartial)
+	require.Len(t, groupsPartial, 4)
+	fsPartial.setFail(groupsPartial[1].Tasks[0].File.FilePath(), injected)
+	beforeFiles := allParquetFiles(t, tblPartial.Location())
+
+	txPartial := tblPartial.NewTransaction()
+	result, err := txPartial.RewriteDataFiles(t.Context(), groupsPartial, table.RewriteDataFilesOptions{
+		PartialProgress: true,
+		MaxCommits:      1,
+		MaxConcurrency:  4,
+	})
+	require.Error(t, err)
+	require.NotNil(t, result)
+	assert.Contains(t, err.Error(), injected.Error())
+	assert.Empty(t, result.CompletedGroups)
+	assert.ElementsMatch(t, beforeFiles, allParquetFiles(t, tblPartial.Location()))
+}
+
+type gateOpenIO struct {
+	iceio.LocalFS
+	mu      sync.Mutex
+	enabled bool
+	entered chan struct{}
+	release chan struct{}
+}
+
+func (g *gateOpenIO) enable() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	g.enabled = true
+	g.entered = make(chan struct{}, 32)
+	g.release = make(chan struct{})
+}
+
+func (g *gateOpenIO) releaseAll() {
+	g.mu.Lock()
+	defer g.mu.Unlock()
+	select {
+	case <-g.release:
+	default:
+		close(g.release)
+	}
+}
+
+func (g *gateOpenIO) Open(name string) (iceio.File, error) {
+	g.mu.Lock()
+	enabled, entered, release := g.enabled, g.entered, g.release
+	g.mu.Unlock()
+	if enabled && strings.Contains(name, "/data/") {
+		select {
+		case <-release:
+		default:
+			select {
+			case entered <- struct{}{}:
+			default:
+			}
+			<-release
+		}
+	}
+
+	return g.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyContextCancel(t *testing.T) {
+	for _, partial := range []bool{false, true} {
+		gate := &gateOpenIO{}
+		tbl := newMaxConcPartitionedTable(t, gate)
+		tbl = addMaxConcPartitions(t, tbl, 8, 1, 10)
+		groups := groupsByPartition(t, tbl)
+		require.Len(t, groups, 8)
+		gate.enable()
+
+		ctx, cancel := context.WithCancel(t.Context())
+		done := make(chan struct{})
+		var rewriteErr error
+		go func() {
+			defer close(done)
+			tx := tbl.NewTransaction()
+			opts := table.RewriteDataFilesOptions{MaxConcurrency: 4}
+			if partial {
+				opts.PartialProgress = true
+				opts.MaxCommits = 1
+			}
+			_, rewriteErr = tx.RewriteDataFiles(ctx, groups, opts)
+		}()
+
+		for range 4 {
+			select {
+			case <-gate.entered:
+			case <-done:
+				t.Fatalf("rewrite finished before 4 groups were in flight, err=%v", rewriteErr)
+			case <-t.Context().Done():
+				t.Fatal("test context done while waiting for groups")
+			}
+		}
+		cancel()
+		gate.releaseAll()
+		<-done
+		require.Error(t, rewriteErr)
+		assert.ErrorIs(t, rewriteErr, context.Canceled)
+		assert.Equal(t, ctx.Err(), rewriteErr)
+	}
+}
+
+type countOpenFile struct {
+	iceio.File
+	owner *countOpenIO
+	once  sync.Once
+}
+
+func (f *countOpenFile) Close() error {
+	err := f.File.Close()
+	f.once.Do(func() {
+		f.owner.mu.Lock()
+		defer f.owner.mu.Unlock()
+		f.owner.cur--
+	})
+
+	return err
+}
+
+type countOpenIO struct {
+	iceio.LocalFS
+	mu   sync.Mutex
+	cur  int
+	peak int
+}
+
+func (c *countOpenIO) Open(name string) (iceio.File, error) {
+	f, err := c.LocalFS.Open(name)
+	if err != nil {
+		return nil, err
+	}
+	if strings.Contains(name, "/data/") {
+		c.mu.Lock()
+		c.cur++
+		if c.cur > c.peak {
+			c.peak = c.cur
+		}
+		c.mu.Unlock()
+		time.Sleep(20 * time.Millisecond)
+
+		return &countOpenFile{File: f, owner: c}, nil
+	}
+
+	return f, nil
+}
+
+func (c *countOpenIO) reset() {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	c.cur = 0
+	c.peak = 0
+}
+
+func (c *countOpenIO) getPeak() int {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	return c.peak
+}
+
+func TestRewriteDataFiles_MaxConcurrencyLimitsInFlight(t *testing.T) {
+	for _, maxConc := range []int{4, 0, 1} {

Review Comment:
   These table cases run in a bare `for` loop, so a `require.*` failure at `maxConc=4` aborts the loop and the 0 and 1 cases never run, and a plain failure won't say which case broke.
   
   I'd wrap each iteration in `t.Run(fmt.Sprintf("maxConc=%d", maxConc), ...)` (same for the `partial=%v` loop in the cancel test). Small thing, just makes failures legible.



##########
table/arrow_scanner.go:
##########
@@ -1242,9 +1242,61 @@ func (as *arrowScan) addTaskProjectedFieldIDs(invariants *arrowScanInvariants, t
 }
 
 type enumeratedRecord struct {
-	Record tblutils.Enumerated[arrow.RecordBatch]
-	Task   tblutils.Enumerated[FileScanTask]
-	Err    error
+	Record  tblutils.Enumerated[arrow.RecordBatch]
+	Task    tblutils.Enumerated[FileScanTask]
+	Err     error
+	credits taskCredits
+}
+
+const maxInFlightTasksPerWorker = 1
+
+type taskCredits chan struct{}
+
+func newTaskCredits() taskCredits {
+	return make(taskCredits, maxInFlightTasksPerWorker)
+}
+
+func (c taskCredits) acquire(ctx context.Context) error {
+	if c == nil {
+		return nil
+	}
+
+	select {
+	case c <- struct{}{}:
+		return nil
+	case <-ctx.Done():
+		return context.Cause(ctx)
+	}
+}
+
+func (c taskCredits) release() {
+	if c != nil {
+		<-c
+	}
+}
+
+type recordSink struct {
+	out     chan<- enumeratedRecord
+	credits taskCredits
+}
+
+func newRecordSink(out chan<- enumeratedRecord) recordSink {
+	return recordSink{out: out, credits: newTaskCredits()}
+}
+
+func (s recordSink) reserve(ctx context.Context) error {
+	return s.credits.acquire(ctx)
+}
+
+func (s recordSink) send(rec enumeratedRecord) {
+	if rec.Record.Last {

Review Comment:
   The credit is only stamped onto the record when `Record.Last` is true, so it's only released downstream on the last batch. That's correct today, but it leans on a subtle invariant: every worker error path (`recordsFromTask` / the delete loaders) returns from the goroutine without ever emitting a `Last`-marked record, so the acquired credit is just abandoned and the worker never calls `reserve()` again on that slot.
   
   The moment someone makes a worker resilient (skip a bad task and continue) that turns into a real deadlock on the next `reserve()`, and nothing here documents or guards the invariant. I'd make release explicit and independent of `Last` (e.g. a `defer` around each task attempt, with send/discard as idempotent no-ops), or at least add a loud comment. wdyt?



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))

Review Comment:
   This is the test carrying the core "concurrent == sequential" claim, but it only compares per-partition row counts, so it'd still pass if a bug swapped rows between files while keeping the counts right.
   
   Since it's the primary evidence for the central claim, I'd make it compare the actual per-partition ID sets (or a hash of the arrow data), not just tallies. Not blocking, but cheap confidence for the one invariant that matters most here.



##########
table/rewrite_data_files_test.go:
##########
@@ -1401,3 +1424,483 @@ func appendEqualityDelete(t *testing.T, tbl *table.Table, equalityFieldIDs []int
 
 	return out
 }
+
+func newMaxConcPartitionedTable(t *testing.T, fs iceio.IO) *table.Table {
+	t.Helper()
+
+	location := filepath.ToSlash(t.TempDir())
+	schema := iceberg.NewSchema(0,
+		iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64, Required: true},
+		iceberg.NestedField{ID: 2, Name: "data", Type: iceberg.PrimitiveTypes.String, Required: false},
+	)
+	spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
+		SourceIDs: []int{2}, FieldID: 1000, Transform: iceberg.IdentityTransform{}, Name: "data",
+	})
+	meta, err := table.NewMetadata(schema, &spec, table.UnsortedSortOrder, location,
+		iceberg.Properties{table.PropertyFormatVersion: "2"})
+	require.NoError(t, err)
+
+	cat := &partialProgressCatalog{metadata: meta}
+
+	return table.New(
+		table.Identifier{"db", "max_conc_test"},
+		meta, location+"/metadata/v1.metadata.json",
+		func(context.Context) (iceio.IO, error) { return fs, nil },
+		cat,
+	)
+}
+
+func addMaxConcPartitions(t *testing.T, tbl *table.Table, partitions, filesPerPartition, rowsPerFile int) *table.Table {
+	t.Helper()
+
+	var nextID int64 = 1
+	for p := range partitions {
+		partition := fmt.Sprintf("p%d", p)
+		for f := range filesPerPartition {
+			ids := make([]int64, rowsPerFile)
+			for r := range rowsPerFile {
+				ids[r] = nextID
+				nextID++
+			}
+			tbl = addPartitionedRowsOnRef(t, tbl, table.MainBranch, fmt.Sprintf("p%d-%d", p, f), partition, ids...)
+		}
+	}
+
+	return tbl
+}
+
+func groupsByPartition(t *testing.T, tbl *table.Table) []table.CompactionTaskGroup {
+	t.Helper()
+
+	tasks, err := tbl.Scan().PlanFiles(t.Context())
+	require.NoError(t, err)
+
+	byPart := make(map[string][]table.FileScanTask)
+	for _, task := range tasks {
+		part, ok := task.File.Partition()[1000].(string)
+		require.True(t, ok)
+		byPart[part] = append(byPart[part], task)
+	}
+	keys := make([]string, 0, len(byPart))
+	for k := range byPart {
+		keys = append(keys, k)
+	}
+	slices.Sort(keys)
+
+	groups := make([]table.CompactionTaskGroup, 0, len(keys))
+	for _, k := range keys {
+		var total int64
+		for _, task := range byPart[k] {
+			total += task.File.FileSizeBytes()
+		}
+		groups = append(groups, table.CompactionTaskGroup{
+			PartitionKey:   k,
+			Tasks:          byPart[k],
+			TotalSizeBytes: total,
+		})
+	}
+
+	return groups
+}
+
+func rowsByPartitionValue(t *testing.T, tbl *table.Table) map[string]int64 {
+	t.Helper()
+
+	_, itr, err := tbl.Scan().ToArrowRecords(t.Context())
+	require.NoError(t, err)
+
+	out := make(map[string]int64)
+	for rec, err := range itr {
+		require.NoError(t, err)
+		idx := rec.Schema().FieldIndices("data")
+		require.NotEmpty(t, idx)
+		col, ok := rec.Column(idx[0]).(*array.String)
+		require.True(t, ok)
+		for i := range int(rec.NumRows()) {
+			out[col.Value(i)]++
+		}
+		rec.Release()
+	}
+
+	return out
+}
+
+func manifestDataPartitions(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var parts []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			part, ok := df.Partition()[1000].(string)
+			require.True(t, ok)
+			parts = append(parts, part)
+		}
+	}
+
+	return parts
+}
+
+func manifestLiveDataPaths(t *testing.T, tbl *table.Table) []string {
+	t.Helper()
+
+	snap := tbl.CurrentSnapshot()
+	require.NotNil(t, snap)
+	fs, err := tbl.FS(t.Context())
+	require.NoError(t, err)
+	manifests, err := snap.Manifests(fs)
+	require.NoError(t, err)
+
+	var paths []string
+	for _, m := range manifests {
+		for e, err := range m.Entries(fs, false) {
+			require.NoError(t, err)
+			if e.Status() == iceberg.EntryStatusDELETED {
+				continue
+			}
+			df := e.DataFile()
+			if df.ContentType() != iceberg.EntryContentData {
+				continue
+			}
+			paths = append(paths, df.FilePath())
+		}
+	}
+
+	return paths
+}
+
+func TestRewriteDataFiles_MaxConcurrencyMatchesSequential(t *testing.T) {
+	tblSeq := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblSeq = addMaxConcPartitions(t, tblSeq, 8, 2, 5)
+	tblConc := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblConc = addMaxConcPartitions(t, tblConc, 8, 2, 5)
+
+	groupsSeq := groupsByPartition(t, tblSeq)
+	groupsConc := groupsByPartition(t, tblConc)
+	require.Len(t, groupsSeq, 8)
+	require.Len(t, groupsConc, 8)
+
+	txSeq := tblSeq.NewTransaction()
+	resSeq, err := txSeq.RewriteDataFiles(t.Context(), groupsSeq, table.RewriteDataFilesOptions{})
+	require.NoError(t, err)
+	committedSeq, err := txSeq.Commit(t.Context())
+	require.NoError(t, err)
+
+	txConc := tblConc.NewTransaction()
+	resConc, err := txConc.RewriteDataFiles(t.Context(), groupsConc, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedConc, err := txConc.Commit(t.Context())
+	require.NoError(t, err)
+
+	assert.Equal(t, resSeq.RewrittenGroups, resConc.RewrittenGroups)
+	assert.Equal(t, resSeq.AddedDataFiles, resConc.AddedDataFiles)
+	assert.Equal(t, resSeq.RemovedDataFiles, resConc.RemovedDataFiles)
+	assert.Equal(t, resSeq.RemovedPositionDeleteFiles, resConc.RemovedPositionDeleteFiles)
+	assert.Equal(t, resSeq.RemovedEqualityDeleteFiles, resConc.RemovedEqualityDeleteFiles)
+	assert.Equal(t, resSeq.RemovedDeletionVectorFiles, resConc.RemovedDeletionVectorFiles)
+	assert.Equal(t, resSeq.BytesBefore, resConc.BytesBefore)
+	assert.Equal(t, 8, resConc.RewrittenGroups)
+	assert.Equal(t, 16, resConc.RemovedDataFiles)
+	assert.Equal(t, 8, resConc.AddedDataFiles)
+
+	assert.Equal(t, rowsByPartitionValue(t, committedSeq), rowsByPartitionValue(t, committedConc))
+	for p := range 8 {
+		assert.Equal(t, int64(10), rowsByPartitionValue(t, committedConc)[fmt.Sprintf("p%d", p)])
+	}
+
+	paths := manifestLiveDataPaths(t, committedConc)
+	require.Len(t, paths, 8)
+	assert.Len(t, map[string]struct{}{paths[0]: {}, paths[1]: {}, paths[2]: {}, paths[3]: {}, paths[4]: {}, paths[5]: {}, paths[6]: {}, paths[7]: {}}, 8)
+	onDisk := allParquetFiles(t, committedConc.Location())
+	for _, p := range paths {
+		assert.Contains(t, onDisk, p)
+	}
+}
+
+func TestRewriteDataFiles_MaxConcurrencyNegativeRejected(t *testing.T) {
+	tbl := newRewriteTestTable(t)
+
+	tx := tbl.NewTransaction()
+	_, err := tx.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+
+	txPartial := tbl.NewTransaction()
+	_, err = txPartial.RewriteDataFiles(t.Context(), nil, table.RewriteDataFilesOptions{PartialProgress: true, MaxConcurrency: -1})
+	require.ErrorIs(t, err, table.ErrInvalidOperation)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyDeterministicOrder(t *testing.T) {
+	tblA := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblA = addMaxConcPartitions(t, tblA, 8, 1, 5)
+	tblB := newMaxConcPartitionedTable(t, iceio.LocalFS{})
+	tblB = addMaxConcPartitions(t, tblB, 8, 1, 5)
+
+	groupsA := groupsByPartition(t, tblA)
+	groupsB := groupsByPartition(t, tblB)
+
+	txA := tblA.NewTransaction()
+	_, err := txA.RewriteDataFiles(t.Context(), groupsA, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedA, err := txA.Commit(t.Context())
+	require.NoError(t, err)
+
+	txB := tblB.NewTransaction()
+	_, err = txB.RewriteDataFiles(t.Context(), groupsB, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.NoError(t, err)
+	committedB, err := txB.Commit(t.Context())
+	require.NoError(t, err)
+
+	orderA := manifestDataPartitions(t, committedA)
+	orderB := manifestDataPartitions(t, committedB)
+	require.Len(t, orderA, 8)
+	require.Len(t, orderB, 8)
+	assert.Equal(t, orderA, orderB)
+	assert.Equal(t, []string{"p0", "p1", "p2", "p3", "p4", "p5", "p6", "p7"}, orderA)
+}
+
+type failOpenIO struct {
+	iceio.LocalFS
+	mu         sync.Mutex
+	failSubstr string
+	failErr    error
+}
+
+func (f *failOpenIO) setFail(substr string, err error) {
+	f.mu.Lock()
+	defer f.mu.Unlock()
+	f.failSubstr = substr
+	f.failErr = err
+}
+
+func (f *failOpenIO) Open(name string) (iceio.File, error) {
+	f.mu.Lock()
+	substr, failErr := f.failSubstr, f.failErr
+	f.mu.Unlock()
+	if substr != "" && strings.Contains(name, substr) {
+		return nil, failErr
+	}
+
+	return f.LocalFS.Open(name)
+}
+
+func TestRewriteDataFiles_MaxConcurrencyGroupFailure(t *testing.T) {
+	injected := errors.New("injected compaction read failure")
+
+	fsAtomic := &failOpenIO{}
+	tblAtomic := newMaxConcPartitionedTable(t, fsAtomic)
+	tblAtomic = addMaxConcPartitions(t, tblAtomic, 4, 1, 5)
+	groupsAtomic := groupsByPartition(t, tblAtomic)
+	require.Len(t, groupsAtomic, 4)
+	fsAtomic.setFail(groupsAtomic[2].Tasks[0].File.FilePath(), injected)
+
+	txAtomic := tblAtomic.NewTransaction()
+	_, err := txAtomic.RewriteDataFiles(t.Context(), groupsAtomic, table.RewriteDataFilesOptions{MaxConcurrency: 4})
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), injected.Error())

Review Comment:
   This half only asserts the error, so it passes whether zero or three groups leaked their output, which is exactly the regression the atomic path is exposed to.
   
   The partial sub-case below captures `beforeFiles := allParquetFiles(...)` and does `assert.ElementsMatch(t, beforeFiles, allParquetFiles(...))` after. I'd add the same before/after check here. Even if it just pins the current (leaky) behavior for now, it makes the trade-off visible and stops a silent regression.



##########
table/arrow_scanner.go:
##########
@@ -2190,6 +2244,7 @@ func createIteratorWithCleanup(ctx context.Context, numWorkers uint, records <-c
 					return
 				}
 
+				enum.credits.release()

Review Comment:
   Small mismatch with the doc claim on `WithCompactionArrowBatchSize`: `release()` fires as soon as the record is popped off the sequenced channel, before `yield` hands it to the caller, so the next worker can start reading its file while the current batch is still being encoded by the consumer.
   
   The real bound is "until the last batch leaves the reorder buffer", which is a touch weaker than the "until consumed" wording. I'd either move `release()` to after `yield` returns, or soften the doc. Not blocking, just want the doc to match the behavior.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#issuecomment-5805516228

   The 5 failing checks fail on main without this PR. go test . -count=1 -run 'TestPartitionTypeFingerprintMatchesAvroConversion|TestManifestEntrySchemaForRejectsInvalidTypesAfterCacheHit' on pristine origin/main at 5d377f1 fails the timestamp_ns and timestamptz_ns subtests and TestManifestEntrySchemaForRejectsInvalidTypesAfterCacheHit, the same failures as here. Both tests and the code they cover came from b46a927 (#2029); this PR touches only table/rewrite_data_files.go, table/rewrite_data_files_test.go and table/rewrite_data_files_bench_test.go, and every table package is green in this run. Nothing to fix on this branch; it needs a green main and a rebase.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4108885236


##########
table/rewrite_data_files.go:
##########
@@ -348,31 +376,32 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
-			return result, err
+	if opts.MaxConcurrentGroups > 1 {
+		results, err := executeCompactionGroups(ctx, t.tbl, groups, opts.GroupOptions, opts.MaxConcurrentGroups)
+		if err != nil {
+			return result, cleanupAtomicRewriteOutputs(ctx, t.tbl, results, err)

Review Comment:
   We clean up here when a group fails, but the terminal `rewrite.Commit(ctx)` around line 432 has no equivalent. If every group writes successfully and then Commit hits a concurrent-modification or catalog conflict (the common atomic-rewrite failure), none of the new files get removed, and `MaxConcurrentGroups` makes it worse since N groups wrote in parallel and one conflict orphans all N groups' output at once. I'd wire `cleanupAtomicRewriteOutputs` into the commit-failure branch too, the same way partial-progress cleans up its own commit failures.



##########
table/rewrite_data_files.go:
##########
@@ -407,6 +436,65 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(ctx context.Context, tbl *Table, results []CompactionGroupResult, cause error) error {
+	fs, err := tbl.fsF(ctx)
+	if err != nil {
+		return errors.Join(cause, fmt.Errorf("open table IO to clean up atomic rewrite outputs: %w", err))
+	}
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}
+	g, gctx := errgroup.WithContext(ctx)
+	g.SetLimit(limit)
+	results := make([]CompactionGroupResult, len(groups))
+	for i, group := range groups {
+		if len(group.Tasks) == 0 {
+			continue
+		}
+		g.Go(func() error {
+			gr, err := ExecuteCompactionGroup(gctx, tbl, group, groupOpts...)
+			results[i] = gr
+
+			return err
+		})
+	}
+	if err := g.Wait(); err != nil {
+		if ctxErr := ctx.Err(); ctxErr != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)) {

Review Comment:
   On concurrent failure the returned error is whichever goroutine finishes first, so a genuine group error can get masked by a sibling's `context.Canceled` (the outer ctx wasn't canceled, so the `ctxErr` guard just above doesn't fire) and the caller loses the real cause. It's also nondeterministic versus sequential's always-first-in-list, so the doc line claiming the result is identical to a sequential run overclaims on the error path. I'd log every non-nil group error via slog so the root cause stays visible, and either pick the lowest-index failing group's error or scope that doc claim to the success path.



##########
table/rewrite_data_files.go:
##########
@@ -407,6 +436,65 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(ctx context.Context, tbl *Table, results []CompactionGroupResult, cause error) error {
+	fs, err := tbl.fsF(ctx)

Review Comment:
   `cleanupAtomicRewriteOutputs` re-derives the fs from `ctx`, and on the cancellation path that ctx is already canceled, so for a real cloud FS (s3/gocloud resolve their config off the context) `fsF(ctx)` can fail before we reach `Remove` and we return having deleted nothing. That's the same leak round-1 was closing, just moved down a level. The partial path avoids it by opening `fs` once before executing any group and passing the handle into cleanup; I'd mirror that here and have this take the already-open `iceio.IO` instead of `(ctx, tbl)`. The stubs don't catch it because every `fsF` just returns `fs, nil` and ignores ctx.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#discussion_r4164725017


##########
table/rewrite_data_files.go:
##########
@@ -401,12 +439,96 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	}
 
 	if err := rewrite.Commit(ctx); err != nil {
-		return result, fmt.Errorf("commit compaction: %w", err)
+		return result, cleanupAtomicRewriteOutputs(fs, applied, fmt.Errorf("commit compaction: %w", err))

Review Comment:
   Wiring cleanup onto `rewrite.Commit` closes part of last round's gap, but `rewrite.Commit` only stages the rewrite onto the transaction. The commit that actually reaches the catalog and can lose a conflict is `txn.Commit` in `Table.RewriteDataFiles` (around line 107), and that path returns the error without removing any outputs. Since `Table.RewriteDataFiles` is the turnkey entry point and the partial branch already cleans up on `ErrCommitFailed`, the atomic action leaks exactly where a real commit conflict lands while partial doesn't. I'd clean up in `Table.RewriteDataFiles` when `txn.Commit` fails with a provably-uncommitted error (`ErrCommitFailed`), mirroring the partial branch.



##########
table/rewrite_data_files.go:
##########
@@ -401,12 +439,96 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	}
 
 	if err := rewrite.Commit(ctx); err != nil {
-		return result, fmt.Errorf("commit compaction: %w", err)
+		return result, cleanupAtomicRewriteOutputs(fs, applied, fmt.Errorf("commit compaction: %w", err))
 	}
 
 	return result, nil
 }
 
+func applyAtomicGroupResult(rewrite *RewriteFiles, result *RewriteResult, stagedDeleteFiles map[string]struct{}, gr CompactionGroupResult) {
+	if len(gr.OldDataFiles) == 0 && len(gr.NewDataFiles) == 0 {
+		return
+	}
+	rewrite.ApplyResult(gr)
+	accumulateGroupMetrics(result, gr)
+	for _, df := range gr.SafePosDeletes {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+	for _, df := range gr.SafeDeletionVectors {
+		stagedDeleteFiles[df.FilePath()] = struct{}{}
+	}
+}
+
+func cleanupAtomicRewriteOutputs(fs iceio.IO, results []CompactionGroupResult, cause error) error {
+	if err := cleanupCompactionOutputs(fs, results); err != nil {
+		return errors.Join(cause, fmt.Errorf("clean up atomic rewrite outputs: %w", err))
+	}
+
+	return cause
+}
+
+func executeCompactionGroups(ctx context.Context, tbl *Table, groups []CompactionTaskGroup, groupOpts []CompactionGroupOption, maxConcurrentGroups int) ([]CompactionGroupResult, error) {
+	if err := ctx.Err(); err != nil {
+		return nil, err
+	}
+	limit := min(maxConcurrentGroups, len(groups))
+	if limit < 1 {
+		limit = 1
+	}
+	var g errgroup.Group
+	g.SetLimit(limit)
+	runCtx, cancelRuns := context.WithCancel(ctx)
+	defer cancelRuns()
+	results := make([]CompactionGroupResult, len(groups))
+	groupErrs := make([]error, len(groups))
+	for i, group := range groups {

Review Comment:
   After `cancelRuns()` fires, this loop keeps dispatching every remaining group; each runs against the canceled runCtx, fails fast, and logs a Warn, so a large plan buries the real error under a wall of context.Canceled warnings. I'd `break` out of the launch loop once `runCtx.Err() != nil`.



##########
table/rewrite_data_files.go:
##########
@@ -348,31 +379,38 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
-			return result, err
-		}
-
-		if len(group.Tasks) == 0 {
-			continue
-		}
+	fs, err := t.tbl.fsF(ctx)

Review Comment:
   Opening fs once up front closes the canceled-open symptom for local backends, but not the leak on cloud. `iceio.IO.Remove` takes no ctx, so blobfs deletes through the ctx the FileIO was built with (`bfs.Delete(bfs.ctx, key)` in `io/gocloud/blobfs/blob.go`), and `fsF(ctx)` threads this request ctx into that field. On the cancel/deadline path, which is the main cleanup trigger, that stored ctx is already dead, so every cleanup `Remove` fails with context.Canceled and the staged outputs stay in object storage. The failure just moved from "open FS" to "delete object."
   
   I'd open the cleanup FS with `context.WithoutCancel(ctx)` and use it only for the Remove calls, or route cleanup through `DeleteFiles(context.WithoutCancel(ctx), paths)` (blobfs threads the per-call ctx there) with a bounded timeout. Same shape in the partial path's `cleanupBatch`. One caveat: the regression test can't catch this today, since `cancelOnDoneFSF` hands back a LocalFS whose Remove ignores ctx, so it passes even if cleanup were a no-op on cloud.



##########
table/rewrite_data_files.go:
##########
@@ -348,31 +379,38 @@ func (t *Transaction) RewriteDataFiles(ctx context.Context, groups []CompactionT
 	rewrite := t.NewRewrite(opts.SnapshotProps)
 	stagedDeleteFiles := make(map[string]struct{})
 
-	for _, group := range groups {
-		if err := ctx.Err(); err != nil {
-			return result, err
-		}
-
-		if len(group.Tasks) == 0 {
-			continue
-		}
+	fs, err := t.tbl.fsF(ctx)
+	if err != nil {
+		return result, fmt.Errorf("open table IO for atomic rewrite: %w", err)

Review Comment:
   This open is now unconditional, so a rewrite with no non-empty groups, which used to return `(result, nil)` without touching IO, now fails here if `fsF` errors. I'd skip the open when there's nothing to compact (gate on `len(groups) == 0`, or open lazily).



##########
table/rewrite_data_files.go:
##########
@@ -217,6 +218,30 @@ type RewriteDataFilesOptions struct {
 	// size, scan concurrency). See the With* helpers returning
 	// [CompactionGroupOption].
 	GroupOptions []CompactionGroupOption
+
+	// MaxConcurrentGroups bounds how many compaction groups run at once.
+	// Zero and one both mean sequential execution, which is the default.
+	// Larger values run [ExecuteCompactionGroup] calls under a bounded
+	// errgroup and apply their results in the original group order, so
+	// manifests and [RewriteResult] are identical to a sequential run.
+	// A failure cancels the groups still running; the returned error
+	// is the lowest-index failure not caused by that cancellation, or
+	// the lowest-index failure when every group was canceled, so
+	// failures match a sequential run. Every group error is logged.
+	// Peak record-pipeline memory is MaxConcurrentGroups times the
+	// per-group bound stated on [WithCompactionArrowBatchSize]:
+	//
+	//	MaxConcurrentGroups x (workers x (rows in the largest task + n) + (recordBatchBufferSize + 2) x n)
+	//
+	// rows, where workers, n and recordBatchBufferSize are the per-group
+	// values. Multiply rows by the average row width in bytes for a byte
+	// estimate. Delete-side memory is outside this bound. File-open
+	// fan-out multiplies too: every group scans with up to
+	// [WithCompactionScanConcurrency] workers, so N groups open about N
+	// times the scan worker count in files at once. Size the two knobs
+	// together against connection and file descriptor limits. Negative
+	// values are rejected with [ErrInvalidOperation].
+	MaxConcurrentGroups int

Review Comment:
   Worth documenting here that under PartialProgress the groups are split into `ceil(len(groups)/MaxCommits)` batches and MaxConcurrentGroups only fans out within a batch. With the default MaxCommits=10, any run of 10 or fewer groups gets one group per batch and this knob does nothing. I'd call out that effective concurrency is `min(MaxConcurrentGroups, ceil(groups/MaxCommits))` so partial-mode callers aren't surprised when the speedup disappears.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] perf(table): run compaction groups concurrently in RewriteDataFiles [iceberg-go]

Posted by "KranzL (via GitHub)" <gi...@apache.org>.
KranzL commented on PR #2041:
URL: https://github.com/apache/iceberg-go/pull/2041#issuecomment-5902136468

   All three round-2 threads are addressed and the branch is rebased onto origin/main at 861b4ca.
   
   6c9a36e wires cleanupAtomicRewriteOutputs into the rewrite.Commit failure branch, opens the table IO once up front with cleanup taking the open handle, and switches error selection to lowest-index-not-canceled with every group failure logged via slog. Cancellation now broadcasts over a plain WithCancel context: with errgroup.WithContext, a still-running sibling scan iterator surfaced the failing group error value through context.Cause instead of context.Canceled, which the new ordering test caught. 1116187 adds the three regression tests; each fails on the pre-fix code and passes after. The rebase drops the data_file_codec.go hunks that duplicated #2052 and keeps only the additive timestamp round-trip coverage, noted in the description.
   
   go test ./table/... -race -count=1 passes (table 37.4s, compaction 13.7s, dv 3.5s, internal 4.6s, substrait 2.1s); golangci-lint reports 0 issues on ./table/...; the benchmark still shows the 1 to 8 speedup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org