You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/03 05:10:04 UTC
[GitHub] [spark] attilapiros opened a new pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
attilapiros opened a new pull request #31719:
URL: https://github.com/apache/spark/pull/31719
### What changes were proposed in this pull request?
Fixing the flaky `SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling` test.
### Why are the changes needed?
Resource profiles in executor PODs allocator are processed in a nondeterministic order.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789727972
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40289/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789692371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135707/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789659088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40285/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789597275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135703/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros edited a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
attilapiros edited a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789602302
I wanted to be 100% sure there is no more flakiness in my test so temporary I have done a small code change locally. I have sorted the map for testing both asc and desc ordering. With this temporary code I have found a bug in a much earlier test (in `SPARK-33288: multiple resource profiles`) which was there even before my SPARK-34361 PR.
So to be on the safe side I modify the production code to have a deterministic ordering for allocating the resource in case of multiple resource profiles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789672705
**[Test build #135707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135707/testReport)** for PR 31719 at commit [`711ea68`](https://github.com/apache/spark/commit/711ea6851c169933a4b6cc4b7d52cae01333dee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789679687
**[Test build #135707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135707/testReport)** for PR 31719 at commit [`711ea68`](https://github.com/apache/spark/commit/711ea6851c169933a4b6cc4b7d52cae01333dee2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros edited a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
attilapiros edited a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789602302
I wanted to be 100% sure that there is no more flakiness in the test. So temporary I have done a small code change and the sorted the map to test both asc and desc ordering. With this temporary code I have found a bug in an earlier test (in `SPARK-33288: multiple resource profiles`) which was there even before my SPARK-34361 PR.
So to be on the safe side I decided to modify the production code to have a deterministic ordering for allocating the resources( in case of multiple resource profiles).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-816597350
closing this as flakiness is not reported by others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789596996
**[Test build #135703 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135703/testReport)** for PR 31719 at commit [`b291d22`](https://github.com/apache/spark/commit/b291d223468bd017a59cf0f9b140c1146a78d08f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789727972
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40289/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789597275
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135703/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789459337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789672705
**[Test build #135707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135707/testReport)** for PR 31719 at commit [`711ea68`](https://github.com/apache/spark/commit/711ea6851c169933a4b6cc4b7d52cae01333dee2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros closed pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
attilapiros closed pull request #31719:
URL: https://github.com/apache/spark/pull/31719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789659088
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40285/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789576575
There is one more place to fix
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789588160
**[Test build #135703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135703/testReport)** for PR 31719 at commit [`b291d22`](https://github.com/apache/spark/commit/b291d223468bd017a59cf0f9b140c1146a78d08f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
attilapiros commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789602302
I wanted to be 100% sure there is no more flakiness in my test so temporary I have done a small code change locally. I have sorted the map for testing both asc and desc ordering. With this temporary code I have found a bug in a much earlier test before my SPARK-34361 PR.
So to be on the safe side I modify the production code to have a deterministic ordering for allocating the resource in case of multiple resource profiles.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789459337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix flaky unit test SPARK-34361: scheduler backend known pods with multiple resource profiles at downscaling
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789588160
**[Test build #135703 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135703/testReport)** for PR 31719 at commit [`b291d22`](https://github.com/apache/spark/commit/b291d223468bd017a59cf0f9b140c1146a78d08f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31719: [SPARK-34361][K8S][FOLLOWUP] Fix potentially flaky unit test with multiple resource profiles in ExecutorPodsAllocatorSuite
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31719:
URL: https://github.com/apache/spark/pull/31719#issuecomment-789692371
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135707/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org