Re: [I] Using the default value of estimate record size at the averageBytesPerRecord() when estimation threshold is less than 0 [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #15586:
URL: https://github.com/apache/hudi/issues/15586#issuecomment-4795449290

   Triage: this appears to be **fixed** on current `master`.
   
   The record size estimation logic was refactored into a pluggable `RecordSizeEstimator` (default: `AverageRecordSizeEstimator`) under HUDI-8985 (#12803), with follow-ups in #14039 and #17524. The current behavior in [`AverageRecordSizeEstimator.averageBytesPerRecord`](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/estimator/AverageRecordSizeEstimator.java) addresses both plans suggested in this issue:
   
   1. **Multi-commit lookback (Plan 1)** — iterates past commits up to `hoodie.record.size.estimator.max.commits` instead of using only the most recent one, so a single tiny commit no longer dominates the estimate.
   2. **Threshold gating + default fallback (Plan 2)** — any commit whose `totalBytesWritten <= recordSizeEstimationThreshold * parquetSmallFileLimit` is skipped, and if no commit qualifies, the estimator falls back to the configured default `hoodie.copyonwrite.record.size.estimate` (`getCopyOnWriteRecordSizeEstimate()`).
   
   If you can still reproduce the original symptom (600–700M outlier files) on a recent release, please share the table config (especially `hoodie.record.size.estimation.threshold`, `hoodie.parquet.small.file.limit`, `hoodie.copyonwrite.record.size.estimate`, and `hoodie.record.size.estimator.max.commits`) and a write-stat snippet so we can re-open. Otherwise, this can likely be closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org