[I] perf(manifest): dataFile.initializeMapData eagerly allocates all column-stats maps on first accessor call [iceberg-go]

["paveon (via GitHub)" <gi...@apache.org>]

paveon opened a new issue, #1142:
URL: https://github.com/apache/iceberg-go/issues/1142

   ### Feature Request / Improvement
   
   ### Description:
   When profiling workloads that read manifest entries (compaction, orphan cleanup), map allocations from `dataFile.initializeMapData()` accounted for ~45% of allocated memory in our use case.
   The root cause is that it uses a single `sync.Once` to convert all 7 column-stats maps plus the
   partition map on the first call to any accessor — even when the caller only needs `Partition()`.
   
   Most code paths outside of table scanning only need partition data, but currently pay the full cost of allocating and populating maps for `ColumnSizes`, `ValueCounts`, `NullValueCounts`, `NaNValueCounts`, `DistinctValueCounts`, `LowerBoundValues`, and `UpperBoundValues`.
   
   Additionally, the underlying `avroColMapToMap` helper does not preallocate the output map, causing repeated rehashing as entries are inserted.
   
   ### Proposed fix:
     1. Split the single `sync.Once` into two — one for partition data, one for column stats — so each accessor only triggers the initialization it needs.
     2. Preallocate the map in `avroColMapToMap` using the known slice length.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org

Re: [I] perf(manifest): dataFile.initializeMapData eagerly allocates all column-stats maps on first accessor call [iceberg-go]

["zeroshade (via GitHub)" <gi...@apache.org>]

zeroshade closed issue #1142: perf(manifest): dataFile.initializeMapData eagerly allocates all column-stats maps on first accessor call
URL: https://github.com/apache/iceberg-go/issues/1142


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org