[jira] [Commented] (SPARK-42479) Log4j2 doesn't works with Spark 3.3.0

["Sean R. Owen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-42479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699041#comment-17699041 ] 

Sean R. Owen commented on SPARK-42479:
--------------------------------------

This is because you're pointing at some local path not visible on the workers. It's quite expected

> Log4j2 doesn't works with Spark 3.3.0
> -------------------------------------
>
>                 Key: SPARK-42479
>                 URL: https://issues.apache.org/jira/browse/SPARK-42479
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, Spark Submit
>    Affects Versions: 3.2.1, 3.3.0
>            Reporter: Pratik Malani
>            Priority: Major
>
> Hi All,
> Was trying to run spark application on the cluster mode using log4j2 and Spark 3.3.0.
> When I run the below spark-submit command, only one worker (out of 3) starts executing the job.
> {code:java}
> // code placeholder
> spark-submit --master spark://spark-master-svc:7077 \
>     --conf spark.cores.max=4 \
>     --conf spark.sql.broadcastTimeout=3600 \
>     --conf spark.executor.cores=1 \
>     --jars /opt/spark/work-dir/<main_jar_name>.jar \
>     --deploy-mode cluster \
>     --class <class_name> \
>     --properties-file /opt/spark/conf/spark-defaults.conf \
>     --conf spark.driver.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
>     --conf spark.executor.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
>     --files "/opt/spark/work-dir/<log4j2>.properties" \
>     /opt/spark/work-dir/<jar_name>.jar /opt/spark/work-dir/application.properties >> /var/log/containers/hourly.log 2>&1 {code}
> It means, on only one worker, I can see the driver logs and the other workers are idle and there are no app or executors logs created on other workers.
> Below is the log4j2.properties file being used.
> {code:java}
> // code placeholder
> rootLogger.level = INFO
> rootLogger.appenderRef.rolling.ref = loggerId
> appender.rolling.type = RollingFile
> appender.rolling.name = loggerId
> appender.rolling.fileName=/var/log/containers/hourly.log
> appender.rolling.filePattern=hourly-.%d{yyyyMMdd}.log.gz
> appender.rolling.layout.type = PatternLayout
> appender.rolling.layout.pattern=%d [%t] %-5p (%F:%L) - %m%n
> appender.rolling.policies.type = Policies
> appender.rolling.policies.size.type = TimeBasedTriggeringPolicy
> appender.rolling.strategy.type = DefaultRolloverStrategy
> appender.rolling.strategy.max = 5
> logger.spark.name = org.apache.spark
> logger.spark.level = WARN
> logger.spark.additivity = false
> logger.spark.repl.SparkIMain$exprTyper.level = INFO
> logger.spark.repl.SparkILoop$SparkILoopInterpreter.level = INFO
> # Settings to quiet third party logs that are too verbose
> logger.jetty.name = org.eclipse.jetty
> logger.jetty.level = WARN
> logger.jetty.util.component.AbstractLifeCycle.level = ERROR
> logger.parquet.name = org.apache.parquet
> logger.parquet.level = ERROR
> logger.kafka.name = org.apache.kafka
> logger.kafka.level = WARN
> logger.kafka.clients.consumer.internals.Fetcher.level=WARN {code}
> All log4j2 jars are included in the Spark home classpath under the jars directory.
>  * log4j-1.2-api-2.17.2.jar
>  * log4j-api-2.17.2.jar
>  * log4j-api-scala_2.12-12.0.jar
>  * log4j-core-2.17.2.jar
>  * log4j-slf4j-impl-2.17.2.jar
> Can you please check and let me know whether I need to add or update anything to start the job in cluster mode supporting log4j2
> Note : Things work fine with log4j1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org