You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by John Fang <xi...@alibaba-inc.com> on 2016/12/09 03:20:45 UTC
how can I set the log configuration file for spark history server ?
./start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/admin/koala/data/versions/0/SPARK/2.0.2/spark-2.0.2-bin-hadoop2.6/logs/spark-admin-org.apache.spark.deploy.history.HistoryServer-1-v069166214.sqa.zmf.out
Then the history will print all log to the XXX.sqa.zmf.out， so i can't limit the file max size. I want limit the size of the log file
Re: how can I set the log configuration file for spark history server ?
Posted by Don Drake <do...@gmail.com>.
You can update $SPARK_HOME/spark-env.sh by setting the environment
variable SPARK_HISTORY_OPTS.
See
http://spark.apache.org/docs/latest/monitoring.html#spark-configuration-options
for options (spark.history.fs.logDirectory) you can set.
There is log rotation built in (by time, not size) to the history server,
you need to enable/configure it.
Hope that helps.
-Don
On Thu, Dec 8, 2016 at 9:20 PM, John Fang <xi...@alibaba-inc.com>
wrote:
> ./start-history-server.sh
> starting org.apache.spark.deploy.history.HistoryServer,
> logging to /home/admin/koala/data/versions/0/SPARK/2.0.2/
> spark-2.0.2-bin-hadoop2.6/logs/spark-admin-org.apache.
> spark.deploy.history.HistoryServer-1-v069166214.sqa.zmf.out
>
> Then the history will print all log to the XXX.sqa.zmf.out， so i can't
> limit the file max size. I want limit the size of the log file
>
--
Donald Drake
Drake Consulting
http://www.drakeconsulting.com/
https://twitter.com/dondrake <http://www.MailLaunder.com/>
800-733-2143
Re: how can I set the log configuration file for spark history server ?
Posted by Don Drake <do...@gmail.com>.
You can update $SPARK_HOME/spark-env.sh by setting the environment
variable SPARK_HISTORY_OPTS.
See
http://spark.apache.org/docs/latest/monitoring.html#spark-configuration-options
for options (spark.history.fs.logDirectory) you can set.
There is log rotation built in (by time, not size) to the history server,
you need to enable/configure it.
Hope that helps.
-Don
On Thu, Dec 8, 2016 at 9:20 PM, John Fang <xi...@alibaba-inc.com>
wrote:
> ./start-history-server.sh
> starting org.apache.spark.deploy.history.HistoryServer,
> logging to /home/admin/koala/data/versions/0/SPARK/2.0.2/
> spark-2.0.2-bin-hadoop2.6/logs/spark-admin-org.apache.
> spark.deploy.history.HistoryServer-1-v069166214.sqa.zmf.out
>
> Then the history will print all log to the XXX.sqa.zmf.out， so i can't
> limit the file max size. I want limit the size of the log file
>
--
Donald Drake
Drake Consulting
http://www.drakeconsulting.com/
https://twitter.com/dondrake <http://www.MailLaunder.com/>
800-733-2143
Re: how can I set the log configuration file for spark history server ?
Posted by Marcelo Vanzin <va...@cloudera.com>.
(-dev)
Just configure your log4j.properties in $SPARK_HOME/conf (or set a
custom $SPARK_CONF_DIR for the history server).
On Thu, Dec 8, 2016 at 7:20 PM, John Fang <xi...@alibaba-inc.com> wrote:
> ./start-history-server.sh
> starting org.apache.spark.deploy.history.HistoryServer, logging to
> /home/admin/koala/data/versions/0/SPARK/2.0.2/spark-2.0.2-bin-hadoop2.6/logs/spark-admin-org.apache.spark.deploy.history.HistoryServer-1-v069166214.sqa.zmf.out
>
> Then the history will print all log to the XXX.sqa.zmf.out， so i can't limit
> the file max size. I want limit the size of the log file
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org