You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wbo4958 (via GitHub)" <gi...@apache.org> on 2024/03/21 07:24:53 UTC
[PR] [TESTS] Refine some gpu fraction calculation tests [spark]
wbo4958 opened a new pull request, #45631:
URL: https://github.com/apache/spark/pull/45631
### What changes were proposed in this pull request?
This PR refines some fractional GPU resource calculation tests.
### Why are the changes needed?
This PR adds more comments to the tests and refines some assertation blocks.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
The existing CI passes.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS][CORE] Refine some GPU fraction calculation tests [spark]
Posted by "Ngone51 (via GitHub)" <gi...@apache.org>.
Ngone51 closed pull request #45631: [SPARK-47498][TESTS][CORE] Refine some GPU fraction calculation tests
URL: https://github.com/apache/spark/pull/45631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS] Refine some GPU fraction calculation tests [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #45631:
URL: https://github.com/apache/spark/pull/45631#discussion_r1535100931
##########
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##########
@@ -2654,18 +2659,21 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext
taskScheduler.submitTasks(lowerTaskSet)
taskScheduler.submitTasks(higherRpTaskSet)
- // only offer the resources to the higher taskset
+ // Initially, offer the available resources to the higher priority task set, where each task
+ // requires 0.7 GPU, so Spark can assign the GPU resources to a maximum of 4 tasks, leaving
+ // 0.3 GPU available for each remaining tasks.
+ // Secondly, try to offer the available resources to the lower priority task set, where each
+ // task requires 0.4 GPU, while only 0.3 gpu of each address is available, so couldn't
+ // offer any resources for the lower tasks.
Review Comment:
LowerTaskSet is indeed submitted before higherRpTaskSet, but there's no resource to offer for lowerTaskSet at that time, lowerTaskSet will be put into the Pool (using FIFO algo by default). After both lowerTaskSet and higherRpTaskSet are in the Pool, the test case begins to offer resources and spark will choose the higher TaskSet (in this case, it is higherTaskSet) to offer. So it should behave like the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS] Refine some GPU fraction calculation tests [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #45631:
URL: https://github.com/apache/spark/pull/45631#discussion_r1537156732
##########
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##########
@@ -2654,18 +2659,21 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext
taskScheduler.submitTasks(lowerTaskSet)
taskScheduler.submitTasks(higherRpTaskSet)
- // only offer the resources to the higher taskset
+ // Initially, offer the available resources to the higher priority task set, where each task
+ // requires 0.7 GPU, so Spark can assign the GPU resources to a maximum of 4 tasks, leaving
+ // 0.3 GPU available for each remaining tasks.
+ // Secondly, try to offer the available resources to the lower priority task set, where each
+ // task requires 0.4 GPU, while only 0.3 gpu of each address is available, so couldn't
+ // offer any resources for the lower tasks.
Review Comment:
@Ngone51 Could you help review it again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS] Refine some GPU fraction calculation tests [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #45631:
URL: https://github.com/apache/spark/pull/45631#issuecomment-2011515195
Hi @Ngone51, Could you help review this PR, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS] Refine some GPU fraction calculation tests [spark]
Posted by "Ngone51 (via GitHub)" <gi...@apache.org>.
Ngone51 commented on code in PR #45631:
URL: https://github.com/apache/spark/pull/45631#discussion_r1535000136
##########
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##########
@@ -2654,18 +2659,21 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext
taskScheduler.submitTasks(lowerTaskSet)
taskScheduler.submitTasks(higherRpTaskSet)
- // only offer the resources to the higher taskset
+ // Initially, offer the available resources to the higher priority task set, where each task
+ // requires 0.7 GPU, so Spark can assign the GPU resources to a maximum of 4 tasks, leaving
+ // 0.3 GPU available for each remaining tasks.
+ // Secondly, try to offer the available resources to the lower priority task set, where each
+ // task requires 0.4 GPU, while only 0.3 gpu of each address is available, so couldn't
+ // offer any resources for the lower tasks.
Review Comment:
It looks `lowerTaskSet` is summitted before `higherRpTaskSet`? So it's actually requires 0.4 GPU per task first and then 0.7 GPU per task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS][CORE] Refine some GPU fraction calculation tests [spark]
Posted by "Ngone51 (via GitHub)" <gi...@apache.org>.
Ngone51 commented on PR #45631:
URL: https://github.com/apache/spark/pull/45631#issuecomment-2021872148
Thanks, merged to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-47498][TESTS] Refine some GPU fraction calculation tests [spark]
Posted by "Ngone51 (via GitHub)" <gi...@apache.org>.
Ngone51 commented on code in PR #45631:
URL: https://github.com/apache/spark/pull/45631#discussion_r1540418049
##########
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
##########
@@ -2654,18 +2659,21 @@ class TaskSchedulerImplSuite extends SparkFunSuite with LocalSparkContext
taskScheduler.submitTasks(lowerTaskSet)
taskScheduler.submitTasks(higherRpTaskSet)
- // only offer the resources to the higher taskset
+ // Initially, offer the available resources to the higher priority task set, where each task
+ // requires 0.7 GPU, so Spark can assign the GPU resources to a maximum of 4 tasks, leaving
+ // 0.3 GPU available for each remaining tasks.
+ // Secondly, try to offer the available resources to the lower priority task set, where each
+ // task requires 0.4 GPU, while only 0.3 gpu of each address is available, so couldn't
+ // offer any resources for the lower tasks.
Review Comment:
I see, thanks for the explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org