You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by QiaoanChen <ka...@gmail.com> on 2014/11/18 09:12:40 UTC
Logging problem in Spark when using Flume Log4jAppender
Hi,
I want to do log aggregation in a Spark standalone mode cluster, using
Apache Flume. But something weird happended. Here are my operations:
(1) Start a flume agent, listening on port 33333. ( flume-conf.properties
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/flume-conf.properties>
)
(2) In log4j.properties, I added a Flume Log4jAppender to make Spark able to
send its log to a flume agent.
( log4j.properties
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/log4j.properties>
)
(3) sbin/start-all.sh.
In this stage, all logs were aggregated into the flume agent and no
exception happened.
(flume-ng-log4jappender version: 1.5.0.1)
(4) bin/spark-submit --class org.apache.examples.SparkPi
../lib/spark-examples-1.1.0-hadoop2.4.0.jar 1
At the beginning, logs were aggregated normally. However, when the
driver need to stop SparkContext sc, it threw some exceptions and the driver
exit abnormally:
sparkPi.log
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi.log>
/====
...
14/11/18 23:06:24 INFO
SparkDeploySchedulerBackend:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- Asking each executor to shut down
14/11/18 23:06:25 INFO
MapOutputTrackerMasterActor:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- MapOutputTrackerActor stopped!
14/11/18 23:06:25 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Selector thread was interrupted!
log4j:ERROR Flume append() failed.
14/11/18 23:06:25 ERROR
ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:96)
- Error in select loop
org.apache.flume.FlumeException: Flume append() failed. Exception follows.
...
====/
I also tried setting log4j.appender.flume.unsafeMode property to true. This
time, the drive finished the job but didn't exit:
sparkPi_snd.log
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n19140/sparkPi_snd.log>
/====
....
14/11/18 23:36:49 WARN
NettyAvroRpcClient:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- Using default maxIOWorkers
log4j:ERROR Cannot Append to Appender! Appender either closed or not setup
correctly!
14/11/18 23:36:49 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Removing ReceivingConnection to ConnectionManagerId(pc215,59272)
14/11/18 23:36:49 ERROR
ConnectionManager:org.apache.spark.Logging$class.logError(Logging.scala:75)
- Corresponding SendingConnection to ConnectionManagerId(pc215,59272) not
found
14/11/18 23:36:49 WARN
ConnectionManager:org.apache.spark.Logging$class.logWarning(Logging.scala:71)
- All connections not cleaned up
14/11/18 23:36:49 INFO
ConnectionManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
ConnectionManager stopped
14/11/18 23:36:49 INFO
MemoryStore:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
MemoryStore cleared
14/11/18 23:36:49 INFO
BlockManager:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
BlockManager stopped
14/11/18 23:36:49 INFO
BlockManagerMaster:org.apache.spark.Logging$class.logInfo(Logging.scala:59)
- BlockManagerMaster stopped
14/11/18 23:36:49 INFO
RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Shutting down remote daemon.
14/11/18 23:36:49 INFO
RemoteActorRefProvider$RemotingTerminator:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Remote daemon shut down; proceeding with flushing remote transports.
14/11/18 23:36:49 INFO
Remoting:akka.event.slf4j.Slf4jLogger$$anonfun$receive$1$$anonfun$applyOrElse$3.apply$mcV$sp(Slf4jLogger.scala:74)
- Remoting shut down
14/11/18 23:36:49 INFO
SparkContext:org.apache.spark.Logging$class.logInfo(Logging.scala:59) -
Successfully stopped SparkContext
====/
I have searched Google with keywords like "Spark log aggregation", "Spark
Flume Log4jAppender", "ERROR Flume append() failed", but I can't figure out
how to solve this problem.
Does anybody has an idea?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Logging-problem-in-Spark-when-using-Flume-Log4jAppender-tp19140.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Logging problem in Spark when using Flume Log4jAppender
Posted by QiaoanChen <ka...@gmail.com>.
Here is the jstack command output, maybe this is helpful:
js.js
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n20255/js.js>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Logging-problem-in-Spark-when-using-Flume-Log4jAppender-tp19140p20255.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org