You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2024/07/26 11:42:28 UTC
[PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
pan3793 opened a new pull request, #47500:
URL: https://github.com/apache/spark/pull/47500
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Currently, structured logging is constantly activated no matter value of `spark.log.structuredLogging.enabled`.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Bug fix, to make `spark.log.structuredLogging.enabled` works with connect server.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No, structured logging is an unreleased feature.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
```
sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=true
```
```
starting org.apache.spark.sql.connect.service.SparkConnectServer, logging to /home/chengpan/Projects/apache-spark/logs/spark-chengpan-org.apache.spark.sql.connect.service.SparkConnectServer-1-HIH-D-25944Z.out
Spark Command: /home/chengpan/.sdkman/candidates/java/current/bin/java -cp hive-jackson/*:/home/chengpan/Projects/apache-spark/conf/:/home/chengpan/Projects/apache-spark/assembly/target/scala-2.13/jars/slf4j-api-2.0.13.jar:/home/chengpan/Projects/apache-spark/assembly/target/scala-2.13/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNN
AMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dderby.connection.requireAuthentication=false org.apache.spark.deploy.SparkSubmit --conf spark.log.structuredLogging.enabled=true --class org.apache.spark.sql.connect.service.SparkConnectServer --name Spark Connect server spark-internal
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
{"ts":"2024-07-26T11:40:40.779Z","level":"WARN","msg":"Your hostname, HIH-D-25944Z, resolves to a loopback address: 127.0.1.1; using 10.219.36.108 instead (on interface enp2s0)","context":{"host":"HIH-D-25944Z","host_port":"127.0.1.1","host_port2":"10.219.36.108","network_if":"enp2s0"},"logger":"Utils"}
{"ts":"2024-07-26T11:40:40.782Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if you need to bind to another address","logger":"Utils"}
uninitLog: true
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
{"ts":"2024-07-26T11:40:41.044Z","level":"INFO","msg":"Starting Spark session.","logger":"SparkConnectServer"}
{"ts":"2024-07-26T11:40:41.067Z","level":"INFO","msg":"Running Spark version 4.0.0-SNAPSHOT","context":{"spark_version":"4.0.0-SNAPSHOT"},"logger":"SparkContext"}
{"ts":"2024-07-26T11:40:41.068Z","level":"INFO","msg":"OS info Linux, 6.5.0-44-generic, amd64","context":{"os_arch":"amd64","os_name":"Linux","os_version":"6.5.0-44-generic"},"logger":"SparkContext"}
{"ts":"2024-07-26T11:40:41.069Z","level":"INFO","msg":"Java version 17.0.8","context":{"java_version":"17.0.8"},"logger":"SparkContext"}
...
```
```
sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
```
```
starting org.apache.spark.sql.connect.service.SparkConnectServer, logging to /home/chengpan/Projects/apache-spark/logs/spark-chengpan-org.apache.spark.sql.connect.service.SparkConnectServer-1-HIH-D-25944Z.out
Spark Command: /home/chengpan/.sdkman/candidates/java/current/bin/java -cp hive-jackson/*:/home/chengpan/Projects/apache-spark/conf/:/home/chengpan/Projects/apache-spark/assembly/target/scala-2.13/jars/slf4j-api-2.0.13.jar:/home/chengpan/Projects/apache-spark/assembly/target/scala-2.13/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNN
AMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dderby.connection.requireAuthentication=false org.apache.spark.deploy.SparkSubmit --conf spark.log.structuredLogging.enabled=false --class org.apache.spark.sql.connect.service.SparkConnectServer --name Spark Connect server spark-internal
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
{"ts":"2024-07-26T11:41:23.459Z","level":"WARN","msg":"Your hostname, HIH-D-25944Z, resolves to a loopback address: 127.0.1.1; using 10.219.36.108 instead (on interface enp2s0)","context":{"host":"HIH-D-25944Z","host_port":"127.0.1.1","host_port2":"10.219.36.108","network_if":"enp2s0"},"logger":"Utils"}
{"ts":"2024-07-26T11:41:23.463Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if you need to bind to another address","logger":"Utils"}
uninitLog: true
Using Spark's default log4j profile: org/apache/spark/log4j2-pattern-layout-defaults.properties
24/07/26 19:41:23 INFO SparkConnectServer: Starting Spark session.
24/07/26 19:41:23 INFO SparkContext: Running Spark version 4.0.0-SNAPSHOT
24/07/26 19:41:23 INFO SparkContext: OS info Linux, 6.5.0-44-generic, amd64
24/07/26 19:41:23 INFO SparkContext: Java version 17.0.8
...
```
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2320499709
I `guess` besides the above scenario, there may be other scenarios that triggered this loading in advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1730635670
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -391,7 +392,8 @@ trait Logging {
private[spark] object Logging {
@volatile private var initialized = false
@volatile private var defaultRootLevel: Level = null
- @volatile private var defaultSparkLog4jConfig = false
+ @volatile private var shouldUseDefaultSparkLog4jConfig = false
+ @volatile private var defaultSparkLog4jConfigApplied = false
Review Comment:
hmm... seems one flag is sufficient, let me change and test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2254241177
hm, this works to be with this change:
```bash
sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
starting org.apache.spark.sql.connect.service.SparkConnectServer, logging to /.../spark/logs/spark-hyukjin.kwon-org.apache.spark.sql.connect.service.SparkConnectServer-1-GF2HX736Y2.out
(python3.11) ➜ spark git:(master) tail -f /.../spark/logs/spark-hyukjin.kwon-org.apache.spark.sql.connect.service.SparkConnectServer-1-GF2HX736Y2.out
```
```
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@70029d2d{/metrics,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@654e6a90{/jobs/job/kill,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@16a5c7e4{/stages/stage/kill,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1e957e2f{/metrics/json,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO Utils: Successfully started service 'org.apache.spark.sql.connect.service.SparkConnectService$' on port 15002.
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4a642e4b{/connect,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3b4086c1{/connect/json,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5eed6dfb{/connect/session,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@551be9f6{/connect/session/json,null,AVAILABLE,@Spark}
24/07/27 16:02:11 INFO SparkConnectServer: Spark Connect server started at: 0:0:0:0:0:0:0:0:15002
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311702580
> > ```
> > sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
> > ```
>
> I used the command `dev/make-distribution.sh --tgz -Phive` to package the latest code.
>
> And without this pr, I manually run `sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false`, and it did use `structuredLogging`, even though `spark.log.structuredLogging.enabled` was specified as `false`:
>
> ```
> tail -100f logs/spark-yangjie01-org.apache.spark.sql.connect.service.SparkConnectServer-1-MacBook-Pro.local.out
>
> Spark Command: /Users/yangjie01/Tools/zulu17/bin/java -cp hive-jackson/*:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/conf/:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/slf4j-api-2.0.16.jar:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED -
-add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dderby.connection.requireAuthentication=false org.apache.spark.deploy.SparkSubmit --conf spark.log.structuredLogging.enabled=false --class org.apache.spark.sql.connect.service.SparkConnectServer --name Spark Connect server spark-internal
> ========================================
> WARNING: Using incubator modules: jdk.incubator.vector
> Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
> {"ts":"2024-08-27T06:42:25.198Z","level":"WARN","msg":"Your hostname, MacBook-Pro.local, resolves to a loopback address: 127.0.0.1; using 172.22.200.254 instead (on interface en0)","context":{"host":"MacBook-Pro.local","host_port":"127.0.0.1","host_port2":"172.22.200.254","network_if":"en0"},"logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.199Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if you need to bind to another address","logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.309Z","level":"INFO","msg":"Starting Spark session.","logger":"SparkConnectServer"}
> {"ts":"2024-08-27T06:42:25.321Z","level":"INFO","msg":"Running Spark version 4.0.0-SNAPSHOT","logger":"SparkContext"}
> {"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"OS info Mac OS X, 14.6.1, aarch64","logger":"SparkContext"}
> {"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"Java version 17.0.12","logger":"SparkContext"}
> {"ts":"2024-08-27T06:42:25.359Z","level":"WARN","msg":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable","logger":"NativeCodeLoader"}
> {"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
> {"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"No custom resources configured for spark.driver.","logger":"ResourceUtils"}
> {"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
> {"ts":"2024-08-27T06:42:25.376Z","level":"INFO","msg":"Submitted application: Spark Connect server","logger":"SparkContext"}
> {"ts":"2024-08-27T06:42:25.383Z","level":"INFO","msg":"Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)","logger":"ResourceProfile"}
> {"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Limiting resource is cpu","logger":"ResourceProfile"}
> {"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Added ResourceProfile id: 0","logger":"ResourceProfileManager"}
> {"ts":"2024-08-27T06:42:25.400Z","level":"INFO","msg":"Changing view acls to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing view acls groups to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls groups to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.402Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: yangjie01 groups with view permissions: EMPTY; users with modify permissions: yangjie01; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.483Z","level":"INFO","msg":"Successfully started service 'sparkDriver' on port 51354.","logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.493Z","level":"INFO","msg":"Registering MapOutputTracker","logger":"SparkEnv"}
> {"ts":"2024-08-27T06:42:25.496Z","level":"INFO","msg":"Registering BlockManagerMaster","logger":"SparkEnv"}
> {"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information","logger":"BlockManagerMasterEndpoint"}
> {"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"BlockManagerMasterEndpoint up","logger":"BlockManagerMasterEndpoint"}
> {"ts":"2024-08-27T06:42:25.504Z","level":"INFO","msg":"Registering BlockManagerMasterHeartbeat","logger":"SparkEnv"}
> {"ts":"2024-08-27T06:42:25.512Z","level":"INFO","msg":"Created local directory at /private/var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/blockmgr-33631b4f-63ad-49a7-8285-f286ff242854","logger":"DiskBlockManager"}
> {"ts":"2024-08-27T06:42:25.519Z","level":"INFO","msg":"Registering OutputCommitCoordinator","logger":"SparkEnv"}
> {"ts":"2024-08-27T06:42:25.564Z","level":"INFO","msg":"Start Jetty 0.0.0.0:4040 for SparkUI","logger":"JettyUtils"}
> {"ts":"2024-08-27T06:42:25.587Z","level":"INFO","msg":"Successfully started service 'SparkUI' on port 4040.","logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls groups to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls groups to: yangjie01","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: yangjie01 groups with view permissions: EMPTY; users with modify permissions: yangjie01; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
> {"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Starting executor ID driver on host 172.22.200.254","logger":"Executor"}
> {"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"OS info Mac OS X, 14.6.1, aarch64","logger":"Executor"}
> {"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Java version 17.0.12","logger":"Executor"}
> {"ts":"2024-08-27T06:42:25.633Z","level":"INFO","msg":"Starting executor with user classpath (userClassPathFirst = false): ''","logger":"Executor"}
> {"ts":"2024-08-27T06:42:25.634Z","level":"INFO","msg":"Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@6fff46bf for default.","logger":"Executor"}
> {"ts":"2024-08-27T06:42:25.641Z","level":"INFO","msg":"Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51355.","logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Server created on 172.22.200.254:51355","logger":"NettyBlockTransferService"}
> {"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy","logger":"BlockManager"}
> {"ts":"2024-08-27T06:42:25.648Z","level":"INFO","msg":"Registering BlockManager BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMaster"}
> {"ts":"2024-08-27T06:42:25.650Z","level":"INFO","msg":"Registering block manager 172.22.200.254:51355 with 434.4 MiB RAM, BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMasterEndpoint"}
> {"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Registered BlockManager BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMaster"}
> {"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Initialized BlockManager: BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManager"}
> {"ts":"2024-08-27T06:42:25.793Z","level":"INFO","msg":"Successfully started service 'org.apache.spark.sql.connect.service.SparkConnectService$' on port 15002.","logger":"Utils"}
> {"ts":"2024-08-27T06:42:25.798Z","level":"INFO","msg":"Spark Connect server started at: 0:0:0:0:0:0:0:0:15002","logger":"SparkConnectServer"}
> ```
After packaging, I didn't perform any other actions before executing `sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false`, except for unpacking the tarball
cc @panbingkun @wayneguow do you have time to help check this issue? Thanks ~ I'm not sure if it's related to some environmental characteristics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311598489
> Please make sure there is no changes in dist/conf/ which may change the logging behavior.
I had checked that, and it definitely has no change. I also checked `env | grep SPARK_`, no unexpected settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1721186702
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -391,7 +392,8 @@ trait Logging {
private[spark] object Logging {
@volatile private var initialized = false
@volatile private var defaultRootLevel: Level = null
- @volatile private var defaultSparkLog4jConfig = false
+ @volatile private var shouldUseDefaultSparkLog4jConfig = false
+ @volatile private var defaultSparkLog4jConfigApplied = false
Review Comment:
Sorry for the late reply.
Not exactly. I found
```
Logging.initializeLogging()
Logging.uninitialize()
Logging.initializeLogging() # this **may** not work properly again
```
To fix the semantic, it introduces a flag to memorize the action of the first call `initializeLogging`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2323163355
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1739647434
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -424,7 +424,6 @@ private[spark] object Logging {
def uninitialize(): Unit = initLock.synchronized {
if (isLog4j2()) {
Review Comment:
unrelated to this pr, but should we additionally check `initialized` here?
```scala
if (initialized && isLog4j2()) {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1717343280
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
ok, I will merge this one once the unused variable is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1716158021
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -391,7 +392,8 @@ trait Logging {
private[spark] object Logging {
@volatile private var initialized = false
@volatile private var defaultRootLevel: Level = null
- @volatile private var defaultSparkLog4jConfig = false
+ @volatile private var shouldUseDefaultSparkLog4jConfig = false
+ @volatile private var defaultSparkLog4jConfigApplied = false
Review Comment:
Note: this is not useful. We can just remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311761655
When there is no configuration (eg: `log4j2.properties`) in `conf`, the above issue will be triggered;
If there is, then this issue does not exist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2316765792
> I still can't reproduce this one latest master branch. I will try on a different laptop and get back here today.
>
> Update: I tried on another laptop and still can't reproduce the issue.
>
> @pan3793 @LuciferYang @panbingkun Are you using the latest master branch?
Yes, I conducted the test on August 27th. The `revision` recorded in the `spark-version-info.properties` file of the packaged Spark client is `9fc1e05224d2b4207885a1d4f822fed6db0cb1a1`, which corresponds to commit
```
commit 9fc1e05224d2b4207885a1d4f822fed6db0cb1a1
Author: Hyukjin Kwon <gu...@apache.org>
Date: Tue Aug 27 14:51:02 2024 +0900
[SPARK-49402][PYTHON] Fix Binder integration in PySpark documentation
```
This was relatively new code on August 27th.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1714796828
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
@gengliangwang do you have better suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311936075
https://github.com/apache/spark/pull/47500#issuecomment-2311482962 https://github.com/apache/spark/pull/47500#issuecomment-2311595579
The difference between these two is that the log of @pan3793 has one additional line of `Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties`, and then it uses `structuredLogging` for printing, more like an issue caused by a classpath ...although I have not yet found the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2252881716
cc @gengliangwang @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2320650889
@LuciferYang Thanks for digging out this issue.
> change ... from val to lazy val. This can avoid the aforementioned unexpected early initialization.
IMO we can not forbid log system early initialization thoroughly, the approach proposed in this PR to reconfigure logging after applying `spark.log.structuredLogging.enabled` makes more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1706450529
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
@pan3793 sorry for the late reply. We should fix this issue for sure. However, explicitly calling `Logging.uninitialize()` here seems weird.
I wonder if we can have a better solution than this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1706456561
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
`Logging.uninitialize()` is designed and was used to re-initialize the logging system
https://github.com/apache/spark/blob/168230a812870b51d84e0a2bd3d10cbba4994eaf/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L960-L963
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #47500: [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled`
URL: https://github.com/apache/spark/pull/47500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1731898620
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -424,7 +424,6 @@ private[spark] object Logging {
def uninitialize(): Unit = initLock.synchronized {
if (isLog4j2()) {
if (defaultSparkLog4jConfig) {
- defaultSparkLog4jConfig = false
Review Comment:
keeping this line and it still works. Shall we keep it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2316876809
> Are you using the latest master branch?
@gengliangwang yes
> ... verified the latest master branch https://github.com/apache/spark/commit/d64d1f76ff28f97db0ad3f3647ae2683e80095a2
@LuciferYang thanks for the help!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2270235943
kindly ping @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2309314933
some of GHA workflows are outdated, rebased on master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311610052
@gengliangwang I notice a difference, seems your log does not have
```
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
```
or
```
Using Spark's default log4j profile: org/apache/spark/org/apache/spark/log4j2-pattern-layout-defaults.properties
```
while my log has, this should indicate 1)that the `initializeLogging` is called by different places with `silent` true and false, or 2) your log4j does not load the spark default configuration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2320658948
> @LuciferYang Thanks for digging out this issue.
>
> > change ... from val to lazy val. This can avoid the aforementioned unexpected early initialization.
>
> IMO we can not forbid logging system early initialization thoroughly, the approach proposed in this PR to reconfigure logging after applying `spark.log.structuredLogging.enabled` makes more sense.
fine to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2323162517
@LuciferYang Thanks a lot for the investigation!
I verified that your assumption is true
```
> InetAddress.getLocalHost
val res0: java.net.InetAddress = hostname/192.168.1.96
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1716158408
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
I will get back to this one in one day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2320426384
In the scenario described by the current PR, I believe the issue is real:
1. The initialization of the `org.apache.spark.internal.config` object precedes the actual effectiveness of the `--conf spark.log.structuredLogging.enabled=false` option from the command line. The former is initialized in `org.apache.spark.deploy.SparkSubmitArguments#loadEnvironmentArguments`, while the latter waits until `org.apache.spark.deploy.SparkSubmit#doSubmit` to execute `Logging.enableStructuredLogging()` or `Logging.disableStructuredLogging()`.
2. The `org.apache.spark.internal.config` object triggers the invocation of the `Utils.localCanonicalHostName()` method during initialization.
https://github.com/apache/spark/blob/0602020eb3b346a8c50ad32eeda4e6dabb70c584/core/src/main/scala/org/apache/spark/internal/config/package.scala#L1149-L1153
3. If the environment variable `SPARK_LOCAL_HOSTNAME` is not set, the `Utils.localCanonicalHostName()` method will call `org.apache.spark.util.Utils#findLocalInetAddress` to initialize `org.apache.spark.util.Utils#localIpAddress`.
https://github.com/apache/spark/blob/0602020eb3b346a8c50ad32eeda4e6dabb70c584/core/src/main/scala/org/apache/spark/util/Utils.scala#L927-L929
https://github.com/apache/spark/blob/0602020eb3b346a8c50ad32eeda4e6dabb70c584/core/src/main/scala/org/apache/spark/util/Utils.scala#L913
https://github.com/apache/spark/blob/0602020eb3b346a8c50ad32eeda4e6dabb70c584/core/src/main/scala/org/apache/spark/util/Utils.scala#L871
4. In the `findLocalInetAddress()` method, if the environment variable `SPARK_LOCAL_IP` is set, the issue described in the current PR will not occur.
5. Otherwise, if `InetAddress.getLocalHost` can obtain the actual IP of the current host instead of `127.0.1.1`, the issue described in the current PR will also not occur.
6. When the return value of `InetAddress.getLocalHost` is `127.0.1.1`, it triggers additional calculations to obtain the actual physical machine IP. At this time, it triggers log printing. However, since `Logging.enableStructuredLogging()` or `Logging.disableStructuredLogging()` has not been called based on `spark.log.structuredLogging.enabled` at this point, `org.apache.spark.internal.Logging#isStructuredLoggingEnabled` will return the default value, which is true. Therefore, it will load `org/apache/spark/log4j2-defaults.properties` to use `StructuredLogging`.
https://github.com/apache/spark/blob/0602020eb3b346a8c50ad32eeda4e6dabb70c584/core/src/main/scala/org/apache/spark/util/Utils.scala#L873-L911
A possible solution I can think of is to change `org.apache.spark.internal.config#DRIVER_HOST_ADDRESS` and `org.apache.spark.internal.config#DRIVER_BIND_ADDRESS` from `val` to `lazy val`. This can avoid the aforementioned unexpected early initialization. I manually tested this solution, and it can restore the logs to normal. However, whether this will cause other side effects, I am currently unable to determine.
Could you help confirm whether `InetAddress.getLocalHost` returns the actual IP address instead of `127.0.0.1` in your testing environment? @HyukjinKwon @gengliangwang
Also, do you have any other ideas for fixing this issue? @HyukjinKwon @gengliangwang @pan3793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2316870317
These days I have some other things to do. I will investigate why some environments load `org/apache/spark/log4j2-defaults.properties` at the weekend, if there is no further conclusion before that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311482962
@gengliangwang issue still exists, verified the latest master branch d64d1f76ff28f97db0ad3f3647ae2683e80095a2
```
$ dev/make-distribution.sh
$ cd dist
$ SPARK_NO_DAEMONIZE=1 sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
```
<img width="1872" alt="image" src="https://github.com/user-attachments/assets/6fb6696a-3e42-4d7e-9f6c-468c6088f8ce">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1730641764
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -391,7 +392,8 @@ trait Logging {
private[spark] object Logging {
@volatile private var initialized = false
@volatile private var defaultRootLevel: Level = null
- @volatile private var defaultSparkLog4jConfig = false
+ @volatile private var shouldUseDefaultSparkLog4jConfig = false
+ @volatile private var defaultSparkLog4jConfigApplied = false
Review Comment:
tested, it works well with one flag too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311595579
@pan3793 I tried and can't reproduce your issue. Please make sure there is no changes in `dist/conf/` which may change the logging behavior.
```
$ dev/make-distribution.sh
$ cd dist
$ SPARK_NO_DAEMONIZE=1 sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
```
<img width="1197" alt="image" src="https://github.com/user-attachments/assets/62eb31e4-2749-4d94-b906-126166ff95e8">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1730622388
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -391,7 +392,8 @@ trait Logging {
private[spark] object Logging {
@volatile private var initialized = false
@volatile private var defaultRootLevel: Level = null
- @volatile private var defaultSparkLog4jConfig = false
+ @volatile private var shouldUseDefaultSparkLog4jConfig = false
+ @volatile private var defaultSparkLog4jConfigApplied = false
Review Comment:
```
$ git grep defaultSparkLog4jConfigApplied
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala: Logging.defaultSparkLog4jConfigApplied = true
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala: @volatile private var defaultSparkLog4jConfigApplied = false
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala: defaultSparkLog4jConfigApplied = false
```
The variable `defaultSparkLog4jConfigApplied` is write-only and does not appear to be used in any conditional checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2311695631
> ```
> sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false
> ```
I used the command `dev/make-distribution.sh --tgz -Phive` to package the latest code.
And without this pr, I manually run `sbin/start-connect-server.sh --conf spark.log.structuredLogging.enabled=false`, and it did use `structuredLogging`, even though `spark.log.structuredLogging.enabled` was specified as false:
```
tail -100f logs/spark-yangjie01-org.apache.spark.sql.connect.service.SparkConnectServer-1-MacBook-Pro.local.out
Spark Command: /Users/yangjie01/Tools/zulu17/bin/java -cp hive-jackson/*:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/conf/:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/slf4j-api-2.0.16.jar:/Users/yangjie01/Tools/4.0/spark-4.0.0-SNAPSHOT-bin-3.4.0/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --a
dd-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dderby.connection.requireAuthentication=false org.apache.spark.deploy.SparkSubmit --conf spark.log.structuredLogging.enabled=false --class org.apache.spark.sql.connect.service.SparkConnectServer --name Spark Connect server spark-internal
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
{"ts":"2024-08-27T06:42:25.198Z","level":"WARN","msg":"Your hostname, MacBook-Pro.local, resolves to a loopback address: 127.0.0.1; using 172.22.200.254 instead (on interface en0)","context":{"host":"MacBook-Pro.local","host_port":"127.0.0.1","host_port2":"172.22.200.254","network_if":"en0"},"logger":"Utils"}
{"ts":"2024-08-27T06:42:25.199Z","level":"WARN","msg":"Set SPARK_LOCAL_IP if you need to bind to another address","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.309Z","level":"INFO","msg":"Starting Spark session.","logger":"SparkConnectServer"}
{"ts":"2024-08-27T06:42:25.321Z","level":"INFO","msg":"Running Spark version 4.0.0-SNAPSHOT","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"OS info Mac OS X, 14.6.1, aarch64","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.322Z","level":"INFO","msg":"Java version 17.0.12","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.359Z","level":"WARN","msg":"Unable to load native-hadoop library for your platform... using builtin-java classes where applicable","logger":"NativeCodeLoader"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"No custom resources configured for spark.driver.","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.375Z","level":"INFO","msg":"==============================================================","logger":"ResourceUtils"}
{"ts":"2024-08-27T06:42:25.376Z","level":"INFO","msg":"Submitted application: Spark Connect server","logger":"SparkContext"}
{"ts":"2024-08-27T06:42:25.383Z","level":"INFO","msg":"Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)","logger":"ResourceProfile"}
{"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Limiting resource is cpu","logger":"ResourceProfile"}
{"ts":"2024-08-27T06:42:25.384Z","level":"INFO","msg":"Added ResourceProfile id: 0","logger":"ResourceProfileManager"}
{"ts":"2024-08-27T06:42:25.400Z","level":"INFO","msg":"Changing view acls to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing view acls groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.401Z","level":"INFO","msg":"Changing modify acls groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.402Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: yangjie01 groups with view permissions: EMPTY; users with modify permissions: yangjie01; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.483Z","level":"INFO","msg":"Successfully started service 'sparkDriver' on port 51354.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.493Z","level":"INFO","msg":"Registering MapOutputTracker","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.496Z","level":"INFO","msg":"Registering BlockManagerMaster","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.503Z","level":"INFO","msg":"BlockManagerMasterEndpoint up","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.504Z","level":"INFO","msg":"Registering BlockManagerMasterHeartbeat","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.512Z","level":"INFO","msg":"Created local directory at /private/var/folders/j2/cfn7w6795538n_416_27rkqm0000gn/T/blockmgr-33631b4f-63ad-49a7-8285-f286ff242854","logger":"DiskBlockManager"}
{"ts":"2024-08-27T06:42:25.519Z","level":"INFO","msg":"Registering OutputCommitCoordinator","logger":"SparkEnv"}
{"ts":"2024-08-27T06:42:25.564Z","level":"INFO","msg":"Start Jetty 0.0.0.0:4040 for SparkUI","logger":"JettyUtils"}
{"ts":"2024-08-27T06:42:25.587Z","level":"INFO","msg":"Successfully started service 'SparkUI' on port 4040.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing view acls groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"Changing modify acls groups to: yangjie01","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.608Z","level":"INFO","msg":"SecurityManager: authentication disabled; ui acls disabled; users with view permissions: yangjie01 groups with view permissions: EMPTY; users with modify permissions: yangjie01; groups with modify permissions: EMPTY; RPC SSL disabled","logger":"SecurityManager"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Starting executor ID driver on host 172.22.200.254","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"OS info Mac OS X, 14.6.1, aarch64","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.631Z","level":"INFO","msg":"Java version 17.0.12","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.633Z","level":"INFO","msg":"Starting executor with user classpath (userClassPathFirst = false): ''","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.634Z","level":"INFO","msg":"Created or updated repl class loader org.apache.spark.util.MutableURLClassLoader@6fff46bf for default.","logger":"Executor"}
{"ts":"2024-08-27T06:42:25.641Z","level":"INFO","msg":"Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51355.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Server created on 172.22.200.254:51355","logger":"NettyBlockTransferService"}
{"ts":"2024-08-27T06:42:25.643Z","level":"INFO","msg":"Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy","logger":"BlockManager"}
{"ts":"2024-08-27T06:42:25.648Z","level":"INFO","msg":"Registering BlockManager BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMaster"}
{"ts":"2024-08-27T06:42:25.650Z","level":"INFO","msg":"Registering block manager 172.22.200.254:51355 with 434.4 MiB RAM, BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMasterEndpoint"}
{"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Registered BlockManager BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManagerMaster"}
{"ts":"2024-08-27T06:42:25.651Z","level":"INFO","msg":"Initialized BlockManager: BlockManagerId(driver, 172.22.200.254, 51355, None)","logger":"BlockManager"}
{"ts":"2024-08-27T06:42:25.793Z","level":"INFO","msg":"Successfully started service 'org.apache.spark.sql.connect.service.SparkConnectService$' on port 15002.","logger":"Utils"}
{"ts":"2024-08-27T06:42:25.798Z","level":"INFO","msg":"Spark Connect server started at: 0:0:0:0:0:0:0:0:15002","logger":"SparkConnectServer"}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1731898620
##########
common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:
##########
@@ -424,7 +424,6 @@ private[spark] object Logging {
def uninitialize(): Unit = initLock.synchronized {
if (isLog4j2()) {
if (defaultSparkLog4jConfig) {
- defaultSparkLog4jConfig = false
Review Comment:
keeping this line and it still works. Shall we keep it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2313195372
I still can't reproduce this one latest master branch. I will try on a different laptop and get back here today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "panbingkun (via GitHub)" <gi...@apache.org>.
panbingkun commented on PR #47500:
URL: https://github.com/apache/spark/pull/47500#issuecomment-2320501472
> I still can't reproduce this one latest master branch. I will try on a different laptop and get back here today.
>
> Update: I tried on another laptop and still can't reproduce the issue.
>
> @pan3793 @LuciferYang @panbingkun Are you using the latest master branch?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1717343280
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
ok, I will merge this one once the unused variable is removed.
Thanks for fixing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled` [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #47500:
URL: https://github.com/apache/spark/pull/47500#discussion_r1716158408
##########
core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:
##########
@@ -85,6 +85,10 @@ private[spark] class SparkSubmit extends Logging {
Logging.disableStructuredLogging()
}
}
+
+ // We should initialize log again after `spark.log.structuredLogging.enabled` effected
+ Logging.uninitialize()
Review Comment:
I will get back to this one in one day.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org