[jira] [Commented] (SPARK-21521) History service requires user is in any group

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099078#comment-16099078 ] 

Marcelo Vanzin commented on SPARK-21521:
----------------------------------------

This smells of a configuration issue... {{root}} is not a special user on HDFS, so running the SHS as root doesn't mean much assuming you're using HDFS.

For HDFS, the event log directory should have "1777" permissions, and the group should be a group to which the SHS user belongs. e.g., in my cluster:

{noformat}
$ hdfs dfs -ls /user/spark
Found 1 items
drwxrwxrwt   - spark spark          0 2017-07-24 13:26 /user/spark/applicationHistory
{noformat}

My SHS runs as user "spark" which belongs to group "spark".

> History service requires user is in any group
> ---------------------------------------------
>
>                 Key: SPARK-21521
>                 URL: https://issues.apache.org/jira/browse/SPARK-21521
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Adrian Bridgett
>
> (Regression cf. 2.0.2)
> We run spark as several users, these write to the history location where the files are saved as those users with permissions of 770 (this is hardcoded in EventLoggingListener.scala).
> The history service runs as root so that it has permissions on these files (see https://spark.apache.org/docs/latest/security.html).
> This worked fine in v2.0.2, however in v2.2.0 the events are being skipped unless I add the root user into each users group at which point they are seen.
> We currently have all acls configuration unset.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org