You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "huangzhir (via GitHub)" <gi...@apache.org> on 2023/04/21 06:56:32 UTC
[GitHub] [kyuubi] huangzhir opened a new pull request, #4749: Fixtest schedulerpool
huangzhir opened a new pull request, #4749:
URL: https://github.com/apache/kyuubi/pull/4749
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
To fix issue https://github.com/apache/kyuubi/issues/4713, a PR https://github.com/apache/kyuubi/pull/4714 was submitted, but it had Flaky test issues. After 50 local tests, it succeeded 38 times and failed 12 times.
This PR addresses the issue of flaky tests.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] cfmcgrady commented on a diff in pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on code in PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#discussion_r1173559242
##########
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/SchedulerPoolSuite.scala:
##########
@@ -101,6 +101,10 @@ class SchedulerPoolSuite extends WithSparkSQLEngine with HiveJDBCTestHelper {
})
threads.shutdown()
threads.awaitTermination(20, SECONDS)
+ // make sure the SparkListener has received the finished events for job1 and job2.
Review Comment:
if the root cause is the event not being received, let's call `KyuubiSparkContextHelper.waitListenerBus()` to make sure there are no more events in the spark event queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on pull request #4749: Fixtest schedulerpool
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#issuecomment-1517367998
Here are the test results for 50 local tests:
Before the modification:
total_time: 823, avg_time: 16
total: 50, success: 38, fail: 12
After the modification:
total_time: 916, avg_time: 18
total: 50, success: 50, fail: 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 closed pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #4749: Fix flaky test issues in SchedulerPoolSuite
URL: https://github.com/apache/kyuubi/pull/4749
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] cfmcgrady commented on pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#issuecomment-1517916689
Good catch! thank you for the fix. @huangzhir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] codecov-commenter commented on pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#issuecomment-1517461116
## [Codecov](https://codecov.io/gh/apache/kyuubi/pull/4749?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#4749](https://codecov.io/gh/apache/kyuubi/pull/4749?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d4558ea) into [master](https://codecov.io/gh/apache/kyuubi/commit/ccacb33c690f4853dab45854dc8063b23c4251a0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ccacb33) will **decrease** coverage by `0.02%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #4749 +/- ##
============================================
- Coverage 58.08% 58.07% -0.02%
Complexity 13 13
============================================
Files 581 581
Lines 32325 32325
Branches 4308 4308
============================================
- Hits 18777 18773 -4
- Misses 11747 11753 +6
+ Partials 1801 1799 -2
```
[see 7 files with indirect coverage changes](https://codecov.io/gh/apache/kyuubi/pull/4749/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] huangzhir commented on a diff in pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "huangzhir (via GitHub)" <gi...@apache.org>.
huangzhir commented on code in PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#discussion_r1173700397
##########
externals/kyuubi-spark-sql-engine/src/test/scala/org/apache/kyuubi/engine/spark/SchedulerPoolSuite.scala:
##########
@@ -101,6 +101,10 @@ class SchedulerPoolSuite extends WithSparkSQLEngine with HiveJDBCTestHelper {
})
threads.shutdown()
threads.awaitTermination(20, SECONDS)
+ // make sure the SparkListener has received the finished events for job1 and job2.
Review Comment:
Thank you for pointing out, KyuubiSparkContextHelper.waitListenerBus is indeed a better idea.The average execution time is reduced by 1 second.
total_time: 882 ,avg_time: 17
total: 50, success: 50, fail: 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [kyuubi] pan3793 commented on pull request #4749: Fix flaky test issues in SchedulerPoolSuite
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #4749:
URL: https://github.com/apache/kyuubi/pull/4749#issuecomment-1517781839
Thanks, merged to master/1.7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org