You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Noritaka Sekiyama (JIRA)" <ji...@apache.org> on 2019/06/17 02:32:00 UTC
[jira] [Updated] (SPARK-28069) Switch log directory from Spark UI
without restarting history server
[ https://issues.apache.org/jira/browse/SPARK-28069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noritaka Sekiyama updated SPARK-28069:
--------------------------------------
Description:
History server polls the directory specified in spark.history.fs.logDirectory, and displays the event logs based on the data included in the directory.
In the Cloud, event logs might be written into different directories from different clusters.
Currently it is not so easy to launch history server for multiple clusters because there is no ways to switch log directory after starting the history server.
If users can switch log directory from Spark UI without restarting history server, it would be beneficial for users who wants to use Cloud storage for event log directory.
Suggested implementation
* History server polls the directory specified in spark.history.fs.logDirectory when it started.
* Add a button of "Switch log directory" at the top page of Web UI.
* The directory which users selected is stored only in memory. (It won't be persistent configuration.)
* Next time when history server started, the directory specified in spark.history.fs.logDirectory is used.
was:
History server polls the directory specified in spark.history.fs.logDirectory, and displays the event logs based on the data included in the directory.
In the Cloud, event logs might be written into different directories from different clusters.
Currently it is not so easy to launch history server for multiple clusters because there is no ways to switch log directory after starting the history server.
If users can switch log directory from Spark UI without restarting history server, it would be beneficial for users who wants to use Cloud storage for event log directory.
> Switch log directory from Spark UI without restarting history server
> --------------------------------------------------------------------
>
> Key: SPARK-28069
> URL: https://issues.apache.org/jira/browse/SPARK-28069
> Project: Spark
> Issue Type: Improvement
> Components: Web UI
> Affects Versions: 2.4.3
> Reporter: Noritaka Sekiyama
> Priority: Minor
>
> History server polls the directory specified in spark.history.fs.logDirectory, and displays the event logs based on the data included in the directory.
> In the Cloud, event logs might be written into different directories from different clusters.
> Currently it is not so easy to launch history server for multiple clusters because there is no ways to switch log directory after starting the history server.
> If users can switch log directory from Spark UI without restarting history server, it would be beneficial for users who wants to use Cloud storage for event log directory.
> Suggested implementation
> * History server polls the directory specified in spark.history.fs.logDirectory when it started.
> * Add a button of "Switch log directory" at the top page of Web UI.
> * The directory which users selected is stored only in memory. (It won't be persistent configuration.)
> * Next time when history server started, the directory specified in spark.history.fs.logDirectory is used.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org