You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2024/12/10 22:54:16 UTC
(spark) branch master updated: [SPARK-50536][CORE] Log downloaded archive file sizes in `SparkContext` and `Executor`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 2093bae154e0 [SPARK-50536][CORE] Log downloaded archive file sizes in `SparkContext` and `Executor`
2093bae154e0 is described below
commit 2093bae154e062d1845083c14c9787f5b65f00b1
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Dec 10 14:54:02 2024 -0800
[SPARK-50536][CORE] Log downloaded archive file sizes in `SparkContext` and `Executor`
### What changes were proposed in this pull request?
This PR aims to log downloaded archive file sizes in `SparkContext` and `Executor`.
### Why are the changes needed?
To improve the UX by showing the file size intuitively in order to help check file health.
### Does this PR introduce _any_ user-facing change?
No, this is a log message.
### How was this patch tested?
Manual check with the Apache Spark test jar file.
- https://github.com/apache/spark/raw/refs/heads/master/data/artifact-tests/smallJar.jar
```
$ ls -al data/artifact-tests/smallJar.jar
-rwxr-xr-x 1 dongjoon staff 787 Dec 3 09:06 data/artifact-tests/smallJar.jar
```
Run the spark-shell and grep `(787 bytes)` from the log message like the following.
```
$ bin/spark-shell --archives https://github.com/apache/spark/raw/refs/heads/master/data/artifact-tests/smallJar.jar -c spark.log.level=DEBUG 2>&1 | grep '787 bytes'
24/12/10 10:05:09 INFO SparkContext: Unpacking an archive https://github.com/apache/spark/raw/refs/heads/master/data/artifact-tests/smallJar.jar (787 bytes) from /private/var/folders/c8/9r5k3d7n19g2m72c86qm08580000gn/T/spark-28ceb587-8376-403e-b364-e675529aeef0/smallJar.jar to /private/var/folders/c8/9r5k3d7n19g2m72c86qm08580000gn/T/spark-d97fc8fc-358e-4044-a904-dfc6823d742d/userFiles-89ff1008-a0d7-4c83-bd9f-9c4e2e830645/smallJar.jar
24/12/10 10:05:09 INFO Executor: Unpacking an archive https://github.com/apache/spark/raw/refs/heads/master/data/artifact-tests/smallJar.jar (787 bytes) from /private/var/folders/c8/9r5k3d7n19g2m72c86qm08580000gn/T/spark-b9020649-cb2b-476b-89f5-5d502f1e0ba8/smallJar.jar to /private/var/folders/c8/9r5k3d7n19g2m72c86qm08580000gn/T/spark-d97fc8fc-358e-4044-a904-dfc6823d742d/userFiles-89ff1008-a0d7-4c83-bd9f-9c4e2e830645/smallJar.jar
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49132 from dongjoon-hyun/SPARK-50536.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
core/src/main/scala/org/apache/spark/SparkContext.scala | 1 +
core/src/main/scala/org/apache/spark/executor/Executor.scala | 1 +
2 files changed, 2 insertions(+)
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index 042179d86c31..20180e142f57 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1878,6 +1878,7 @@ class SparkContext(config: SparkConf) extends Logging {
if (uri.getFragment != null) uri.getFragment else source.getName)
logInfo(
log"Unpacking an archive ${MDC(LogKeys.PATH, path)}" +
+ log" (${MDC(LogKeys.BYTE_SIZE, source.length)} bytes)" +
log" from ${MDC(LogKeys.SOURCE_PATH, source.getAbsolutePath)}" +
log" to ${MDC(LogKeys.DESTINATION_PATH, dest.getAbsolutePath)}")
Utils.deleteRecursively(dest)
diff --git a/core/src/main/scala/org/apache/spark/executor/Executor.scala b/core/src/main/scala/org/apache/spark/executor/Executor.scala
index c299f38526ae..f1087b695a7d 100644
--- a/core/src/main/scala/org/apache/spark/executor/Executor.scala
+++ b/core/src/main/scala/org/apache/spark/executor/Executor.scala
@@ -1210,6 +1210,7 @@ private[spark] class Executor(
if (sourceURI.getFragment != null) sourceURI.getFragment else source.getName)
logInfo(
log"Unpacking an archive ${LogMDC(ARCHIVE_NAME, name)}" +
+ log" (${LogMDC(BYTE_SIZE, source.length)} bytes)" +
log" from ${LogMDC(SOURCE_PATH, source.getAbsolutePath)}" +
log" to ${LogMDC(DESTINATION_PATH, dest.getAbsolutePath)}")
Utils.deleteRecursively(dest)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org