You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Saketh Kurnool (Jira)" <ji...@apache.org> on 2024/07/12 22:28:00 UTC

[jira] [Updated] (FLINK-35830) Large failed bulk request can result in TM OOM

     [ https://issues.apache.org/jira/browse/FLINK-35830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Saketh Kurnool updated FLINK-35830:
-----------------------------------
    Description: 
The {{BulkResponse}} object contains a single {{Throwable}} per failing document in the bulk request. The connector currently loops through the failures, combines them into a single exception via suppression, and throws it. In cases where the bulk size is very large (>10,000 responses), the size of the resulting stack trace is so large that serializing it causes the TM to OOM. Attached is the heap dump visualization of a TM that OOM'ed with a failing bulk size of >10,000.

I have mitigated this issue in my local fork of the OS connector by only suppressing exceptions from each bulk response with unique root causes - this way, we can avoid massively nested stack traces where the root cause of every failure is the exact same. NOTE that this proposed fix does *not* mitigate the unlikely case in which every failing document in a very large {{BulkResponse}} has a different root cause. I believe this is acceptable judging by how infrequently this would occur, but it is worth revisiting in the future if it becomes a problem.

@opensearch connector community: let me know if you think this fix would be valuable - I'm happy to open a PR for this upstream!

  was:
The {{BulkResponse}} object contains a single {{Throwable}} per failing document in the bulk request. The connector currently loops through the failures, combines them into a single exception via suppression, and throws it. In cases where the bulk size is very large (>10,000 responses), the size of the resulting stack trace is so large that serializing it causes the TM to OOM. Attached is the heap dump visualization of a TM that OOM'ed with a failing bulk size of >10,000.

I have mitigated this issue in my local fork of the OS connector by only suppressing exceptions from a bulk response with unique root causes - this way, we can avoid massively nested stack traces where the root cause of every failure is the exact same. NOTE that this proposed fix does *not* mitigate the unlikely case in which every failing document in a very large {{BulkResponse}} has a different root cause. I believe this is acceptable judging by how infrequently this would occur, but it is worth revisiting in the future if it becomes a problem.

@opensearch connector community: let me know if you think this fix would be valuable - I'm happy to open a PR for this upstream!


> Large failed bulk request can result in TM OOM
> ----------------------------------------------
>
>                 Key: FLINK-35830
>                 URL: https://issues.apache.org/jira/browse/FLINK-35830
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Opensearch
>            Reporter: Saketh Kurnool
>            Priority: Major
>         Attachments: Screenshot 2024-07-12 at 2.50.44 PM.png
>
>
> The {{BulkResponse}} object contains a single {{Throwable}} per failing document in the bulk request. The connector currently loops through the failures, combines them into a single exception via suppression, and throws it. In cases where the bulk size is very large (>10,000 responses), the size of the resulting stack trace is so large that serializing it causes the TM to OOM. Attached is the heap dump visualization of a TM that OOM'ed with a failing bulk size of >10,000.
> I have mitigated this issue in my local fork of the OS connector by only suppressing exceptions from each bulk response with unique root causes - this way, we can avoid massively nested stack traces where the root cause of every failure is the exact same. NOTE that this proposed fix does *not* mitigate the unlikely case in which every failing document in a very large {{BulkResponse}} has a different root cause. I believe this is acceptable judging by how infrequently this would occur, but it is worth revisiting in the future if it becomes a problem.
> @opensearch connector community: let me know if you think this fix would be valuable - I'm happy to open a PR for this upstream!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)