You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by guoxiaolongzte <gi...@git.apache.org> on 2017/08/02 08:18:44 UTC

[GitHub] spark pull request #18815: [SPARK-21609][WEB-UI]In the Master ui add "log di...

GitHub user guoxiaolongzte opened a pull request:

    https://github.com/apache/spark/pull/18815

    [SPARK-21609][WEB-UI]In the Master ui add "log directory" display, is conducive to users to quickly find the log directory path.

    ## What changes were proposed in this pull request?
    In the Master ui add "log directory" display, is conducive to users to quickly find the log directory path.
    
    In the spark application development process, we not only view the executor log and driver log, but also to see the master log and worker log, but the current UI will not show the master and worker log path, resulting in the user is not very clear to find the log path. So, I add "log directory" display.
    
    
    ## How was this patch tested?
    
    manual tests
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/guoxiaolongzte/spark SPARK-21609

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/18815.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18815
    
----
commit 32e182e6c48f43e53ecbcd2f759363961c7e73ec
Author: guoxiaolong <gu...@zte.com.cn>
Date:   2017-08-02T08:14:53Z

    [SPARK-21609]In the Master ui add "log directory" display, is conducive to users to quickly find the log directory path.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by ajbozarth <gi...@git.apache.org>.

Github user ajbozarth commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    I'm not a fan of this, I remember talking about surfacing this a while ago and we decided that it was a bad idea, but we may have been talking about surfacing the log not the path. Either way only admins would be able to access that log so displaying it on the UI doesn't add anything in my opinion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    @srowen 
    Are you sure that the master and worker logs can be stored in hdfs?
    Spark's master and worker logs are generated by log4j.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    I don't know if this is that important, so not sure if it's worth much code. Would people generally be able to access the logs anyway? is there a security issue if I can see the masters logs and open them up to everyone? I don't know that this is a good idea.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    @srowen Help review the code,Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    It's not an http URI though right? it's a path. I'm missing why this is browseable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    1.The master process information is as follows：
     /usr/java8/jdk/bin/java -cp /opt/ZDH/parcels/lib/spark_gxl/conf/:/opt/ZDH/parcels/lib/spark_gxl/jars/*:/opt/ZDH/parcels/lib/spark_gxl/conf/hdfs_conf/:/opt/ZDH/parcels/lib/spark_gxl/conf/yarn_conf -Dspark.ui.allowFramingFrom=ALLOW-FROM -DLOG_LEVEL=INFO -DROLE_NAME=master -DLOG_FILE=/data1/zdh/spark/logs/spark-root-master-zdh167.log -Xmx1024M org.apache.spark.deploy.master.Master --host zdh167 --port 7078 --webui-port 8081
    
    Log4j can not write log files to hdfs.
    Event log is through the hdfs client to write the log to hdfs.
    
    2.Currently only the log path is displayed in the UI. If you can, the details will be displayed in the future, and you can view.
    e.g.
    ![11](https://user-images.githubusercontent.com/26266482/28916845-1f737782-7876-11e7-8f9e-3323cedf5082.png)
    
    ![12](https://user-images.githubusercontent.com/26266482/28916863-2abea922-7876-11e7-9c81-94a1c3712af5.png)
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by ajbozarth <gi...@git.apache.org>.

Github user ajbozarth commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Ok then I'm really confused, if the logs we're talking about can already be viewed in the ui why do we need to display their location on the system?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Yes, I am also considering this security issue. If this only shows a path, only authorized people can log on to the spark cluster lunix server for log view, which is relatively safe.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    This won't necessarily be available to people viewing the UI? it could be an HDFS location


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    I think we should close this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Yes, it is not  an http URI ,at present only a path to tell the user log path where to facilitate the lunix server query related log information.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Ok. That master, worker log and Executor log can be displayed in the WEB UI?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Why is the Executor log can be displayed in the WEB UI?
    
    Similarly, I think master, worker log and Executor log is the same. They can be displayed in the WEB UI.
    
    ![image](https://user-images.githubusercontent.com/26266482/28950185-5fada808-78f4-11e7-9b42-e84dbed1f809.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by jiangxb1987 <gi...@git.apache.org>.

Github user jiangxb1987 commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Yea my concern is whether it is vulnerable to show the log path, maybe system admin want to keep that private.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Hm, I thought so, even in standalone mode, but I don't know.
    OK, if it's a file URI, is that going to be browseable?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by srowen <gi...@git.apache.org>.

Github user srowen commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    OK, then how can this work? this won't render as a usable link.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #18815: [SPARK-21609][WEB-UI]In the Master ui add "log directory...

Posted by guoxiaolongzte <gi...@git.apache.org>.

Github user guoxiaolongzte commented on the issue:

    https://github.com/apache/spark/pull/18815
  
    Ok, i will work it, just like hbase. Show master and worker logs, and provide online viewing. Is this okay?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org