You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "hamilton-earthscope (via GitHub)" <gi...@apache.org> on 2025/11/11 15:50:23 UTC
[PR] perf(table): optimize partitioned write throughput [iceberg-go]
hamilton-earthscope opened a new pull request, #622:
URL: https://github.com/apache/iceberg-go/pull/622
# Partitioned Write Optimizations
## Summary
This PR delivers significant performance improvements to the partitioned write throughput in the Iceberg table writer. Through a series of iterative optimizations, we achieved substantial gains in write performance, reduced memory allocations, and improved overall efficiency.
## Performance Results
Note: the following results are using the latest commit on `main` of arrow-go (https://github.com/apache/arrow-go/commit/3160eef9c227d94db67bfaf5225a2d6c1f48bc76)
The following benchmarks were conducted on Apple M3 Max (darwin/arm64) for the new `BenchmarkPartitionedWriteThroughput` test with 2.5M rows per write operation:
### Incremental Improvements
| Change | Time/op | Δ Time | records/sec | Δ records/sec | allocs/op | Δ allocs/op |
|--------|---------|---------|-------------|---------------|-----------|-------------|
| Base | 2.35 s | - | 1,065,115 | - | 60,076,290 | - |
| Change 1 | 1.49 s | -36.5% | 1,677,802 | +57.5% | 35,076,376 | -41.6% |
| Change 2 | 1.21 s | -18.7% | 2,064,629 | +23.1% | 25,076,562 | -28.5% |
| Change 3 | 1.16 s | -4.5% | 2,161,545 | +4.7% | 22,576,588 | -10.0% |
| Change 4 | 1.07 s | -7.4% | 2,334,700 | +8.0% | 20,076,480 | -11.1% |
| Change 5 | 654.1 ms | -38.9% | 3,821,892 | +63.7% | 12,577,119 | -37.4% |
### Overall Improvement (Base → Change 5)
**2.5M Records per write:**
| Metric | Base | Change 5 | Improvement |
|--------|------|----------|-------------|
| **Time/op** | 2.35 s | 654.1 ms | **-72.1%** ⚡ |
| **records/sec** | 1,065,115 | 3,821,892 | **+258.9%** 🚀 |
| **allocs/op** | 60,076,290 | 12,577,119 | **-79.1%** 💪 |
**100K Records per write:**
| Metric | Base | Change 5 | Improvement |
|--------|------|----------|-------------|
| **Time/op** | 221.3 ms | 137.7 ms | **-37.8%** |
| **records/sec** | 451,853 | 726,085 | **+60.7%** |
| **allocs/op** | 2,472,792 | 573,012 | **-76.8%** |
## Impact
These optimizations significantly improve the performance of partitioned writes in Iceberg tables, making the library more efficient for high-throughput data ingestion scenarios. The improvements scale well with larger workloads, as demonstrated by the dramatic gains in the 2.5M record benchmark results.
## Testing
All optimizations were validated through comprehensive benchmarking on darwin/arm64 (Apple M3 Max). The improvements are consistent across different record volumes (100K, 500K, and 2.5M records).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): optimize partitioned write throughput [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #622:
URL: https://github.com/apache/iceberg-go/pull/622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] perf(table): optimize partitioned write throughput [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #622:
URL: https://github.com/apache/iceberg-go/pull/622#discussion_r2529114279
##########
table/partitioned_fanout_writer.go:
##########
@@ -210,22 +211,103 @@ func (p *partitionedFanoutWriter) getPartitionMap(record arrow.RecordBatch) (map
transformedLiteral := sourceField.Transform.Apply(iceberg.Optional[iceberg.Literal]{Valid: true, Val: val})
if transformedLiteral.Valid {
partitionRec[i] = transformedLiteral.Val.Any()
- partitionValues[sourceField.FieldID] = transformedLiteral.Val.Any()
} else {
- partitionRec[i], partitionValues[sourceField.FieldID] = nil, nil
+ partitionRec[i] = nil
}
} else {
- partitionRec[i], partitionValues[partitionFieldsInfo[i].fieldID] = nil, nil
+ partitionRec[i] = nil
}
}
- partitionKey := p.partitionPath(partitionRec)
- partVal := partitionMap[partitionKey]
- partVal.rows = append(partitionMap[partitionKey].rows, row)
- partVal.partitionValues = partitionValues
- partitionMap[partitionKey] = partVal
+
+ // Get or create partition info for this partition key
+ partVal := partitionMap.getOrCreate(partitionRec, partitionFieldsInfo)
+ partVal.rows = append(partVal.rows, row)
+ }
+
+ return partitionMap.collectPartitions(p.partitionSpec, p.schema), nil
+}
+
+// partitionMapNode represents a simple tree structure for storing partitionInfo.
+//
+// Each key is the partition value at that level of the tree, and the key hierarchy
+// is in the order of the partition spec.
+// The value is either a *partitionMapNode or a *partitionInfo.
+type partitionMapNode struct {
+ children map[any]any
+ leafCount int
+}
+
+func newPartitionMapNode() *partitionMapNode {
+ return &partitionMapNode{
+ children: make(map[any]any),
+ leafCount: 0,
+ }
+}
+
+// getOrCreate navigates the tree and returns the partitionInfo for the given partition key,
+// creating nodes along the way if they don't exist
+func (n *partitionMapNode) getOrCreate(partitionRec partitionRecord, fieldInfo []struct {
+ sourceField *iceberg.PartitionField
+ fieldID int
+},
+) *partitionInfo {
+ // Navigate through all but the last partition field
+ node := n
+ i := 0
+ for ; i < len(partitionRec)-1; i++ {
+ val, ok := node.children[partitionRec[i]]
+ if !ok {
+ newNode := newPartitionMapNode()
+ node.children[partitionRec[i]] = newNode
+ node = newNode
+ } else {
+ node = val.(*partitionMapNode)
+ }
+ }
Review Comment:
can we just use `for _, part := range partitionRec[:len(partitionRec)-1] {` ?
Then below for the last key you can just use `lastKey := partitionRec[len(partitionRec)-1]`
This would simplify the the loop and logic
##########
table/partitioned_fanout_writer.go:
##########
Review Comment:
Given the additional cases we added above, can we reduce the cases here at all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org