You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@texera.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2026/04/28 07:13:40 UTC

[PR] [codex] Fix Python internal marker replay during reconfiguration [texera]

Yicong-Huang opened a new pull request, #4547:
URL: https://github.com/apache/texera/pull/4547

   ## Summary
   - restore consume-on-read semantics for Python `current_internal_marker`
   - stop clearing the marker a second time after replaying internal channel markers
   - prevent stale internal `EndChannel` markers from being observed again during reconfiguration propagation
   
   ## Root Cause
   This regression was introduced by `#4424` (`ef66190f22`). That change made `get_internal_marker()` return `current_internal_marker` without consuming it, and moved marker cleanup to the end of the main-loop replay path.
   
   For Python source operators, the internal `EndChannel` marker can stay live across the pause and reconfiguration window. When the reconfiguration ECM is processed, the stale internal marker is still visible and gets replayed again, which corrupts end-of-stream handling and hangs the workflow.
   
   This change restores the previous consume-on-read behavior so the internal marker is observed exactly once.
   
   Fixes #4545.
   
   ## Validation
   - `WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec`
   - `WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec -- -z "be able to modify a python UDF worker in workflow"`
   - `WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec -- -z "propagate reconfiguration through a source operator in workflow"`
   - `WorkflowExecutionService/testOnly org.apache.texera.amber.engine.e2e.ReconfigurationSpec -- -z "be able to modify two python UDFs in workflow"`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(amber): Python internal marker replay during reconfiguration [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on PR #4547:
URL: https://github.com/apache/texera/pull/4547#issuecomment-4334539099

   > Do you think the comment in [#4424 (comment)](https://github.com/apache/texera/pull/4424#discussion_r3139794598) is valid?
   
   I replied there. I think the comment is not valid, and the true issue in $4424 is not about `switch_context`. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(amber): Python internal marker replay during reconfiguration [texera]

Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang merged PR #4547:
URL: https://github.com/apache/texera/pull/4547


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] fix(amber): Python internal marker replay during reconfiguration [texera]

Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on PR #4547:
URL: https://github.com/apache/texera/pull/4547#issuecomment-4333134996

   Do you think the comment in https://github.com/apache/texera/pull/4424#discussion_r3139794598 is valid?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org