You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/11/28 14:25:27 UTC
[spark] branch branch-3.3 updated: [SPARK-41254][YARN] bugfix wrong usage when check YarnAllocator.rpIdToYarnResource key existence
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 090bebd6a63 [SPARK-41254][YARN] bugfix wrong usage when check YarnAllocator.rpIdToYarnResource key existence
090bebd6a63 is described below
commit 090bebd6a63fdd69b14d08c459fd5bd2301948e4
Author: John Caveman <se...@gmail.com>
AuthorDate: Mon Nov 28 08:25:00 2022 -0600
[SPARK-41254][YARN] bugfix wrong usage when check YarnAllocator.rpIdToYarnResource key existence
### What changes were proposed in this pull request?
bugfix, a misuse of ConcurrentHashMap.contains causing map YarnAllocator.rpIdToYarnResource always updated
### Why are the changes needed?
It causing duplicated log during yarn resource allocation and unnecessary object creation and gc
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing tests
Closes #38790 from CavemanIV/SPARK-41254.
Authored-by: John Caveman <se...@gmail.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
(cherry picked from commit bccfe5bca600b3091ea93b4c5d6437af8381973f)
Signed-off-by: Sean Owen <sr...@gmail.com>
---
.../src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
index a85b7174673..16cae4810e4 100644
--- a/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
+++ b/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala
@@ -278,7 +278,7 @@ private[yarn] class YarnAllocator(
// if a ResourceProfile hasn't been seen yet, create the corresponding YARN Resource for it
private def createYarnResourceForResourceProfile(rp: ResourceProfile): Unit = synchronized {
- if (!rpIdToYarnResource.contains(rp.id)) {
+ if (!rpIdToYarnResource.containsKey(rp.id)) {
// track the resource profile if not already there
getOrUpdateRunningExecutorForRPId(rp.id)
logInfo(s"Resource profile ${rp.id} doesn't exist, adding it")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org