You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2022/11/30 19:47:57 UTC
[spark] branch branch-3.3 updated: [SPARK-41327][CORE] Fix `SparkStatusTracker.getExecutorInfos` by switch On/OffHeapStorageMemory info
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new a94dd1820d6 [SPARK-41327][CORE] Fix `SparkStatusTracker.getExecutorInfos` by switch On/OffHeapStorageMemory info
a94dd1820d6 is described below
commit a94dd1820d65e6280941047dbb4abb15bf429bc3
Author: Lingyun Yuan <yl...@gmail.com>
AuthorDate: Wed Nov 30 11:47:15 2022 -0800
[SPARK-41327][CORE] Fix `SparkStatusTracker.getExecutorInfos` by switch On/OffHeapStorageMemory info
### What changes were proposed in this pull request?
This PR aims to fix `SparkStatusTracker.getExecutorInfos` to return a correct `on/offHeapStorageMemory`.
### Why are the changes needed?
`SparkExecutorInfoImpl` used the following parameter order.
https://github.com/apache/spark/blob/54c57fa86906f933e089a33ef25ae0c053769cc8/core/src/main/scala/org/apache/spark/StatusAPIImpl.scala#L42-L45
SPARK-20659 introduced a bug with wrong parameter order at Apache Spark 2.4.0.
- https://github.com/apache/spark/pull/20546/files#diff-7daca909d33ff8e9b4938e2b4a4aaa1558fbdf4604273b9e38cce32c55e1508cR118-R121
### Does this PR introduce _any_ user-facing change?
Yes.
### How was this patch tested?
Manually review.
Closes #38843 from ylybest/master.
Lead-authored-by: Lingyun Yuan <yl...@gmail.com>
Co-authored-by: ylybest <11...@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 388824c448804161b076507f0f39ef0596e0a0bf)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
core/src/main/scala/org/apache/spark/SparkStatusTracker.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala b/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
index 37e673cd8c7..22dc1d056ec 100644
--- a/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
+++ b/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala
@@ -114,10 +114,10 @@ class SparkStatusTracker private[spark] (sc: SparkContext, store: AppStatusStore
port,
cachedMem,
exec.activeTasks,
- exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
exec.memoryMetrics.map(_.usedOnHeapStorageMemory).getOrElse(0L),
- exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L),
- exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L))
+ exec.memoryMetrics.map(_.usedOffHeapStorageMemory).getOrElse(0L),
+ exec.memoryMetrics.map(_.totalOnHeapStorageMemory).getOrElse(0L),
+ exec.memoryMetrics.map(_.totalOffHeapStorageMemory).getOrElse(0L))
}.toArray
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org