[jira] [Created] (HUDI-9331) Incorrect memory estimation for CDC block flushing can lead to OOM

["Moran (Jira)" <ji...@apache.org>]

Moran created HUDI-9331:
---------------------------

             Summary: Incorrect memory estimation for CDC block flushing can lead to OOM
                 Key: HUDI-9331
                 URL: https://issues.apache.org/jira/browse/HUDI-9331
             Project: Apache Hudi
          Issue Type: Bug
          Components: cdc
            Reporter: Moran


When writing CDC (Change Data Capture) log, Hudi accumulates records in memory until the total estimated size reaches {{{}maxBlockSize{}}}. The flush condition is based on the formula:
{code:java}
numOfCDCRecordsInMemory.get() * averageCDCRecordSize >= maxBlockSize {code}
However, the value of averageCDCRecordSize is only estimated once, during the first write of CDC data
{code:java}
if (cdcData.isEmpty()) {
    averageCDCRecordSize = sizeEstimator.sizeEstimate(payload);
} {code}
This approach can lead to underestimation of memory usage. For instance, if the first CDC record is relatively small but subsequent records are much larger, the estimated average size remains inaccurately low. As a result, the number of records in memory can grow far beyond what would actually fit in maxBlockSize, potentially leading to OutOfMemoryError (OOM) before the flush is triggered.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)