You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/09 03:46:51 UTC
[GitHub] [spark] 19Serhii99 opened a new pull request, #38574: [SPARK-41060] [K8S] Made the spark submitter generate new names for driver and executor config maps
19Serhii99 opened a new pull request, #38574:
URL: https://github.com/apache/spark/pull/38574
### What changes were proposed in this pull request?
There's a problem with submitting spark jobs to K8s cluster: the library generates and reuses the same name for config maps (for drivers and executors). Ideally, for each job 2 config maps should created: for a driver and an executor. However, the library creates only one driver config map for all jobs (in some cases it generates only one executor map for all jobs). So, if I run 5 jobs, then only one driver config map will be generated and used for every job. During those runs we experience issues when deleting pods from the cluster: executors pods are endlessly created and immediately terminated overloading cluster resources.
This problem occurs because of the **KubernetesClientUtils** class in which we have **configMapNameExecutor** and **configMapNameDriver** as constants. It seems to be incorrect and should be urgently fixed. I've prepared some changes for review to fix the issue (tested in the cluster of our project).
### Why are the changes needed?
To make the spark submitter generate new names for driver and executor config maps to let jobs complete successfully, not overloading cluster resources.
### Does this PR introduce _any_ user-facing change?
Executors pods should stop being generated and terminated endlessly.
### How was this patch tested?
Modified the unit tests in the module. Tested with the unit tests and by submitting jobs to the K8S cluster of our project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] 19Serhii99 commented on pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by GitBox <gi...@apache.org>.
19Serhii99 commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1326984005
cc @dongjoon-hyun @Yikun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] commented on pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1464712402
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by GitBox <gi...@apache.org>.
holdenk commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1332802078
The CI failure is in Kube (e.g. `[info] 22/11/10 13:25:54 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources`) can you take a look @19Serhii99 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1310144467
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] 19Serhii99 commented on pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by GitBox <gi...@apache.org>.
19Serhii99 commented on PR #38574:
URL: https://github.com/apache/spark/pull/38574#issuecomment-1311956108
Need help with fixing the integration tests. I did not expect they would fail as I had replaced the constants with the methods and that's it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] github-actions[bot] closed pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #38574: [SPARK-41060][K8S] Fix generating driver and executor Config Maps
URL: https://github.com/apache/spark/pull/38574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org