You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@texera.apache.org by "Yicong-Huang (via GitHub)" <gi...@apache.org> on 2025/10/12 18:22:35 UTC
[I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Yicong-Huang opened a new issue, #3880:
URL: https://github.com/apache/texera/issues/3880
### What happened?
Flaky test: https://github.com/apache/texera/actions/runs/18447656995
### How to reproduce?
N/A
### Version
1.1.0-incubating (Pre-release/Master)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
### Code of Conduct
- [x] I agree to follow Apache Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang closed issue #3880: org.apache.amber.engine.e2e.DataProcessingSpec is flaky
URL: https://github.com/apache/texera/issues/3880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-4291127148
The test case failures no longer appear thanks to the retry mechanism. But we do not have a permanent fix until we can solve #3950.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-3395227912
- [ ]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu closed issue #3880: org.apache.amber.engine.e2e.DataProcessingSpec is flaky
URL: https://github.com/apache/texera/issues/3880
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "chenlica (via GitHub)" <gi...@apache.org>.
chenlica commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-4294002854
@Xiao-zhen-Liu Let's do a discussion on that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "aglinxinyuan (via GitHub)" <gi...@apache.org>.
aglinxinyuan commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-3420439104
> Thanks @Xiao-zhen-Liu for the investigations. I agree with your assessment. Could you please create new task issues for action items you listed?
We can try sub-issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-3420428687
Thanks @Xiao-zhen-Liu for the investigations. I agree with your assessment. Could you please create new task issues for action items you listed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-3420404322
There are two kinds of flaky test behaviors, both related to e2e tests.
- One is for `DataProcessingSpec`. This test suite contains 10 test cases, and each sets a 1-minute timeout. There is a chance on CI environment that one of the the 10 test cases may time out.
- The other is for `PauseSpec` . This is appeared in [PR 3913](https://github.com/apache/texera/actions/runs/18610057938/job/53066773439?pr=3913). The test cases in PauseSpec do not have timeouts internally, and seems to cause the test job to hang there until it reaches Github CI’s 6-hour timeout.
**Cause**: I think the causes of both behaviors are the same. When we run these e2e tests, we execute a workflow and wait for controller to send workflow’s `COMPLETED` status. Somehow in the CI environment there is a chance this message is not sent (or not received) and causes the wait to either hang (for `PauseSpec` ) or trigger our internal timeout (for `DataProcessingSpec`).
**Reproducibility**: This seems to only happen in the CI environment. @aglinxinyuan mentioned he tested locally 10,000 times and could not reproduce this timeout issue.
**Origin**: Regarding the origin of this issue, I think it may be much earlier than recently, possibly since the creation of these test cases. It happens infrequently (maybe 1 in 50 CI runs) and usually rerunning the CI solves the problem. I found in March this year there was a [6-hour timeout](https://github.com/apache/texera/actions/runs/13732392069) in master-branch’s CI run, and the other flaky CI failure may be dated as early as [Sep 2024](https://github.com/apache/texera/actions/runs/10806203712) (earlier runs have been garbage-collected).
**Short-term Solution**: As this problem may be very hard to reproduce or debug, I suggest for all these test cases, we keep enforcing the 1-minute internal timeout but add a retry mechanism, as rerunning the test case always solves the issue. This will stabilize the CIs.
**Permenant Solution**: We should try to reproduce this issue, possibly in an Ubuntu environment, and locate the root cause of the `COMPLETED` status not being sent for these test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Xiao-zhen-Liu (via GitHub)" <gi...@apache.org>.
Xiao-zhen-Liu commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-4291106862
This should be fixed with #4179.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] org.apache.amber.engine.e2e.DataProcessingSpec is flaky [texera]
Posted by "Yicong-Huang (via GitHub)" <gi...@apache.org>.
Yicong-Huang commented on issue #3880:
URL: https://github.com/apache/texera/issues/3880#issuecomment-4286540448
@Xiao-zhen-Liu is this fixed? There is not PR linked on deployment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@texera.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org