You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2026/07/09 04:05:00 UTC
[jira] [Resolved] (SPARK-52566) Logging Level is not getting overrided
[ https://issues.apache.org/jira/browse/SPARK-52566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-52566.
----------------------------------
Resolution: Invalid
Resolving as Invalid — this is a usage/how-to question rather than a specific Spark defect or actionable change. Usage questions are best directed to user@spark.apache.org (https://spark.apache.org/community.html) or Stack Overflow (tag apache-spark). Findings from triage: Verified against apache/master Logging.scala that this is a support/configuration question, not a Spark defect. Spark's initializeLogging (common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:343) only applies its own default profile (log4j2-defaults.properties, rootLogger.level=info) when islog4j2DefaultConfigured() is true (line 472) — which requires the user to have essentially NO real Log4j2 config (empty appenders, or a single ERROR appender whose active config is literally Log4j2's built-in DefaultConfiguration class). Any user-provided config file yields a different Confi
Please reopen with a concrete reproducer or a specific proposed change if this is actually a bug or an actionable improvement.
> Logging Level is not getting overrided
> ---------------------------------------
>
> Key: SPARK-52566
> URL: https://issues.apache.org/jira/browse/SPARK-52566
> Project: Spark
> Issue Type: Question
> Components: Java API, Spark Core
> Affects Versions: 3.4.0
> Environment: Grails - 4.0.12
> Reporter: Anuradha Bhan
> Priority: Major
> Labels: Java, Spark, grails
>
> Grails Project is using Log4j2. Logging level is set to ERROR using Configurator before spark session is started. Even then INFO level logs are getting printed by spark. It is picking up the default log properties inside the spark-core jar.
>
> Code snippet :
>
> Configurator.setLevel("org.apache.spark", Level.ERROR)
> Logger logger = (Logger) LogManager.getLogger("org.apache.spark");
> System.out.println("Logger instance level: " + logger.getLevel());
> logger.debug("This DEBUG should not appear");
> logger.info("This INFO should not appear");
> logger.error("This ERROR should appear");
> SparkSession spark = null;
> StructType schema = createParquetSchema()
> spark = SparkSession
> .builder()
> .appName("XXXX")
> .config("spark.master", "local")
> .config("spark.io.compression.codec", "snappy")
> .config("spark.io.compression.snappy.blockSize", blockSize)
> .getOrCreate();
>
>
> The output comes out to be :
> Logger instance level: DEBUG
> This DEBUG should not appear
> This INFO should not appear
> This ERROR should appear
> Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
> 25/06/24 18:30:43 WARN Utils: Your hostname, AnuradhaB resolves to a loopback address: 127.0.1.1; using <IPAddress> instead (on interface wlp0s20f3)
> 25/06/24 18:30:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
> 25/06/24 18:30:43 INFO SparkContext: Running Spark version 3.4.0
> 25/06/24 18:30:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 25/06/24 18:30:43 INFO ResourceUtils: ==============================================================
> 25/06/24 18:30:43 INFO ResourceUtils: No custom resources configured for spark.driver.
> 25/06/24 18:30:43 INFO ResourceUtils: ==============================================================
> 25/06/24 18:30:43 INFO SparkContext: Submitted application: TestABC
> 25/06/24 18:30:43 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
> 25/06/24 18:30:43 INFO ResourceProfile: Limiting resource is cpu
> 25/06/24 18:30:43 INFO ResourceProfileManager: Added ResourceProfile id: 0
> 25/06/24 18:30:43 INFO SecurityManager: Changing view acls to: anuradha
> 25/06/24 18:30:43 INFO SecurityManager: Changing modify acls to: anuradha
> 25/06/24 18:30:43 INFO SecurityManager: Changing view acls groups to:
> 25/06/24 18:30:43 INFO SecurityManager: Changing modify acls groups to:
> 25/06/24 18:30:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: anuradha; groups with view permissions: EMPTY; users with modify permissions: anuradha; groups with modify permissions: EMPTY
> 25/06/24 18:30:43 INFO Utils: Successfully started service 'sparkDriver' on port 38691.
> 25/06/24 18:30:43 INFO SparkEnv: Registering MapOutputTracker
> 25/06/24 18:30:43 INFO SparkEnv: Registering BlockManagerMaster
> 25/06/24 18:30:43 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> 25/06/24 18:30:43 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> 25/06/24 18:30:43 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 25/06/24 18:30:43 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-abf8fb69-c6a4-40e5-bd43-ad0ccb47e759
> 25/06/24 18:30:43 INFO MemoryStore: MemoryStore started with capacity 4.1 GiB
> 25/06/24 18:30:44 INFO SparkEnv: Registering OutputCommitCoordinator
> 25/06/24 18:30:44 INFO JettyUtils: Start Jetty 0.0.0.0:4040 for SparkUI
> 25/06/24 18:30:44 INFO Utils: Successfully started service 'SparkUI' on port 4040.
> 25/06/24 18:30:44 INFO Executor: Starting executor ID driver on host <IPAddress>
> 25/06/24 18:30:44 INFO Executor: Starting executor with user classpath (userClassPathFirst = false): ''
> 25/06/24 18:30:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42551.
> 25/06/24 18:30:44 INFO NettyBlockTransferService: Server created on <IPAddress>:42551
> 25/06/24 18:30:44 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
> 25/06/24 18:30:44 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, <IPAddress>, 42551, None)
> 25/06/24 18:30:44 INFO BlockManagerMasterEndpoint: Registering block manager <IPAddress>:42551 with 4.1 GiB RAM, BlockManagerId(driver, <IPAddress>, 42551, None)
> 25/06/24 18:30:44 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, <IPAddress>, 42551, None)
> 25/06/24 18:30:44 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, <IPAddress>, 42551, None)
>
>
> What else could be done to set the logging level to ERROR
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org