You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "weiqingy (via GitHub)" <gi...@apache.org> on 2026/01/29 22:25:38 UTC

[PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

weiqingy opened a new pull request, #509:
URL: https://github.com/apache/flink-agents/pull/509

   <!--
   * Thank you very much for contributing to Flink Agents.
   * Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc.
   -->
   
   <!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. -->
   Linked issue: #508 
   
   ### Purpose of change
   
   <!-- What is the purpose of this change? -->
     Fix a crash that occurs when restoring a Python async action from checkpoint.                        
                                                                                                          
     **Root cause:** Python coroutines cannot be serialized. When a checkpoint captures state while an    
     async action is in progress (e.g., waiting for LLM response), the coroutine object is lost. On       
     restore, the awaitable reference is `None`, causing:                                                 
     AttributeError: 'NoneType' object has no attribute 'send'                                            
                                                                                                          
     **Fix:**                                                                                             
     - Detect `None` awaitable in `PythonActionExecutor.callPythonAwaitable()`                            
     - Throw `AwaitableLostException` to signal the awaitable was lost                                    
     - `PythonGeneratorActionTask` catches this and re-executes the action from the beginning             
     - Durable execution cache ensures already-completed calls are skipped   
   
   ### Tests
   
   <!-- How is this change verified? -->
     - Manual testing: Run `react_agent_example.py`, trigger LLM timeout, verify job recovers instead of  
     crashing                                                                                             
     - No unit test added - these classes depend on Pemja (Python interpreter) which is difficult to mock;
      the fix is better validated via e2e testing    
   ### API
   
   <!-- Does this change touches any public APIs? -->
   No public API changes. 
   ### Documentation
   
   <!-- Do not remove this section. Check the proper box only. -->
   
   - [ ] `doc-needed` <!-- Your PR changes impact docs -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-included` <!-- Your PR already contains the necessary documentation updates -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan merged PR #509:
URL: https://github.com/apache/flink-agents/pull/509


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan commented on PR #509:
URL: https://github.com/apache/flink-agents/pull/509#issuecomment-3822295149

   @xintongsong Could you take a look at this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on PR #509:
URL: https://github.com/apache/flink-agents/pull/509#issuecomment-3822092229

   > @weiqingy , Thanks for identifying this bug and submitting the fix! The issue you found is real and important. After reviewing the PR, I made some adjustments to align the Python async handling with how we handle Java continuations. The current approach uses a transient map in the operator (similar to continuationContexts for Java), which makes the checkpoint restore detection more explicit and consistent across both languages. I've pushed a fixup commit with these changes. Could you please take a look and let me know what you think?
   
   Thanks for the quick review and changes, @Sxnan! It LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan commented on PR #509:
URL: https://github.com/apache/flink-agents/pull/509#issuecomment-3821927681

   @weiqingy , Thanks for identifying this bug and submitting the fix! The issue you found is real and important.
   After reviewing the PR, I made some adjustments to align the Python async handling with how we handle Java continuations. The current approach uses a transient map in the operator (similar to continuationContexts for Java), which makes the checkpoint restore detection more explicit and consistent across both languages.
   I've pushed a fixup commit with these changes. Could you please take a look and let me know what you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "weiqingy (via GitHub)" <gi...@apache.org>.
weiqingy commented on PR #509:
URL: https://github.com/apache/flink-agents/pull/509#issuecomment-3821571737

   Hey @Sxnan, wondering if there’s a way to re-run the failed CI? It seems I don’t have access to trigger it. This looks like a pre-existing flaky test, not related to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [runtime] Fix Python awaitable lost during checkpoint restore [flink-agents]

Posted by "Sxnan (via GitHub)" <gi...@apache.org>.
Sxnan commented on PR #509:
URL: https://github.com/apache/flink-agents/pull/509#issuecomment-3822564648

   Ported to release-0.2 in 5d9fb77


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org