You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@amoro.apache.org by "slfan1989 (via GitHub)" <gi...@apache.org> on 2026/04/05 03:07:38 UTC

[I] [Improvement]: Implement heap-based flush mechanism for SortedPosDeleteWriter to prevent OOM [amoro]

slfan1989 opened a new issue, #4166:
URL: https://github.com/apache/amoro/issues/4166

   ### Search before asking
   
   - [x] I have searched in the [issues](https://github.com/apache/amoro/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### What would you like to be improved?
   
   Currently, `SortedPosDeleteWriter` only flushes buffered position deletes based on a record count threshold. There is a `TODO` comment in the code indicating the need for a heap memory-based flush policy:
   
   ```
   // TODO Flush buffer based on the policy that checking whether whole heap memory size exceed the
   // threshold.
   if (records >= recordsNumThreshold) {
     flushDeletes();
   }
   ```
   
   **Problem**: When processing large-scale position deletes, the in-memory buffer in `SortedPosDeleteWriter` can grow unbounded (if record threshold is set very high or to `Long.MAX_VALUE`), potentially causing OutOfMemoryError (OOM) issues, especially in memory-constrained environments.
   
   **Current behavior**:
   - Only flushes when record count reaches **recordsNumThreshold**
   - No protection against heap memory pressure
   - Can lead to OOM when processing large delete operations
   
   ### How should we improve?
   
   Implement a heap memory-based flush mechanism with the following features:
   
   ### 1. New table properties:
   - `pos-delete.flush.heap.ratio (default: 0.8)` - Heap usage ratio threshold to trigger flush
   - `pos-delete.flush.records (default: Long.MAX_VALUE)` - Record count threshold
   - `pos-delete.flush.heap.min-records (default: 1000)` - Minimum records before heap-based flush kicks in
   
   ### 2. Implementation details:
   - Add `HeapUsageProvider` interface to monitor JVM heap usage
   - Implement `shouldFlushByHeap()` method to check if heap usage exceeds threshold
   - Modify flush logic to: `if (records >= recordsNumThreshold || shouldFlushByHeap())`
   - Ensure backward compatibility through constructor overloads
   
   ### 3. Safety guards:
   - Prevent frequent small flushes with minimum record count
   - Allow disabling heap-based flush by setting invalid ratio (≤0 or ≥1)
   - Non-intrusive monitoring (no forced GC)
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Subtasks
   
   _No response_
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement]: Implement heap-based flush mechanism for SortedPosDeleteWriter to prevent OOM [amoro]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] closed issue #4166: [Improvement]: Implement heap-based flush mechanism for SortedPosDeleteWriter to prevent OOM
URL: https://github.com/apache/amoro/issues/4166


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement]: Implement heap-based flush mechanism for SortedPosDeleteWriter to prevent OOM [amoro]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #4166:
URL: https://github.com/apache/amoro/issues/4166#issuecomment-5963422467

   This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Improvement]: Implement heap-based flush mechanism for SortedPosDeleteWriter to prevent OOM [amoro]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #4166:
URL: https://github.com/apache/amoro/issues/4166#issuecomment-6243524568

   This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org