Need help regarding logging / log4j.properties

[Debabrata Ghosh <ma...@gmail.com>]

Greetings All !

I needed some help in obtaining the application logs but I am really
confused where it's currently located. Please allow me to explain my
problem:
1. I am running the Spark application (written in Java) in a Hortonworks
Data Platform Hadoop cluster

2. My spark-submit command is the following:
spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain
--master yarn --packages
org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0
--deploy-mode client --files
job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml
./spark-job.jar --settings=job-configuration.properties
log4j-spark.properties --conf
"spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
--conf
"spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"

3. Content of the log4.properties is the following:
# Set everything to be logged to the console
log4j.rootCategory=ALL, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
%c{1}: %m%n

 # Set the default spark-shell log level to WARN. When running the
spark-shell, the
# log level for this class is used to overwrite the root logger's log
level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ALL
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

 # SPARK-9183: Settings to avoid annoying messages when looking up
nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

#Any custom class debug
log4j.logger.com.vmeg.code=DEBUG

4. Inside the spark application, I am having plenty of logger functions and
one example is like this:
logger.info("No dataset recieved for this batch");

Question: My question is where is the application log files being written ?
I have checked in the yarn logs but couldn't find the messages I have
written in the java file.
Request your help please as I am little confused and know that there is
something very silly which I am missing.

Thanks in advance !

Debu

Re: Need help regarding logging / log4j.properties

[Roland Johann <ro...@phenetic.io.INVALID>]

Hi Debu,

you need to define spark config properties before the jar file path at
spark-submit. Everything after the jar path will be passed as arguments to
your application.

Best Regards

Debabrata Ghosh <ma...@gmail.com> schrieb am Do. 31. Okt. 2019 um
03:26:

> Greetings All !
>
> I needed some help in obtaining the application logs but I am really
> confused where it's currently located. Please allow me to explain my
> problem:
> 1. I am running the Spark application (written in Java) in a Hortonworks
> Data Platform Hadoop cluster
>
> 2. My spark-submit command is the following:
> spark-submit --class com.ibm.spark.sparkkafkamapper.ServiceMapperMain
> --master yarn --packages org.apache.kafka:kafka-clients:0.10.0.1,org.apache.spark:spark-streaming-kafka-0-10_2.11:2.3.0
> --deploy-mode client --files
> job-configuration.properties,log4j.properties,/usr/hdp/current/spark2-client/conf/hive-site.xml
> ./spark-job.jar --settings=job-configuration.properties
> log4j-spark.properties --conf
> "spark.driver.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
> --conf
> "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties"
>
> 3. Content of the log4.properties is the following:
> # Set everything to be logged to the console
> log4j.rootCategory=ALL, console
> log4j.appender.console=org.apache.log4j.ConsoleAppender
> log4j.appender.console.target=System.err
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{1}: %m%n
>
>  # Set the default spark-shell log level to WARN. When running the
> spark-shell, the
> # log level for this class is used to overwrite the root logger's log
> level, so that
> # the user can have different defaults for the shell and regular Spark
> apps.
> log4j.logger.org.apache.spark.repl.Main=WARN
>
> # Settings to quiet third party logs that are too verbose
> log4j.logger.org.spark_project.jetty=ALL
> log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
> log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
> log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
> log4j.logger.org.apache.parquet=ERROR
> log4j.logger.parquet=ERROR
>
>  # SPARK-9183: Settings to avoid annoying messages when looking up
> nonexistent UDFs in SparkSQL with Hive support
> log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
> log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
>
> #Any custom class debug
> log4j.logger.com.vmeg.code=DEBUG
>
> 4. Inside the spark application, I am having plenty of logger functions
> and one example is like this:
> logger.info("No dataset recieved for this batch");
>
> Question: My question is where is the application log files being written
> ? I have checked in the yarn logs but couldn't find the messages I have
> written in the java file.
> Request your help please as I am little confused and know that there is
> something very silly which I am missing.
>
> Thanks in advance !
>
> Debu
>
-- 


*Roland Johann*Software Developer/Data Engineer

*phenetic GmbH*
Lütticher Straße 10, 50674 Köln, Germany

Mobil: +49 172 365 26 46
Mail: roland.johann@phenetic.io
Web: phenetic.io

Handelsregister: Amtsgericht Köln (HRB 92595)
Geschäftsführer: Roland Johann, Uwe Reimann