You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Adrian Bridgett (JIRA)" <ji...@apache.org> on 2017/07/24 20:41:00 UTC
[jira] [Comment Edited] (SPARK-21521) History service requires user
is in any group
[ https://issues.apache.org/jira/browse/SPARK-21521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099090#comment-16099090 ]
Adrian Bridgett edited comment on SPARK-21521 at 7/24/17 8:40 PM:
------------------------------------------------------------------
Hmm, I didn't check that (it's actually just a local filesystem in this case, but I take the point that it's accessed via the Hadoop FS API).
Not sure why it's changed behaviour in 2.2.0 compared to 2.0.2 though.
Just checked access via hadoop (using hdfs command line) and it's certainly still able to access it:
{noformat}
~# id
uid=0(root) gid=0(root) groups=0(root),997(airflow)
~# ls -l /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
-rwxrwx--- 1 ubuntu ubuntu 13982 Jul 24 17:57 /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
~# hdfs dfs -cat /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
..(stuff)
{noformat}
was (Author: abridgett):
Hmm, I didn't check that (it's actually just a local filesystem in this case, but I take the point that it's accessed via the Hadoop FS API).
Not sure why it's changed behaviour in 2.2.0 compared to 2.0.2 though.
Just checked access via hadoop (using hdfs command line) and it's certainly still able to access it:
{noformat}
~# id
uid=0(root) gid=0(root) groups=0(root),997(airflow)
~# ls -l /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
-rwxrwx--- 1 ubuntu ubuntu 13982 Jul 24 17:57 /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4
~# hdfs dfs -cat /var/log/spark/events/a5bab156-b4c2-41a2-93f2-11ba78c99c6e-12221.lz4id
..(stuff)
{noformat}
> History service requires user is in any group
> ---------------------------------------------
>
> Key: SPARK-21521
> URL: https://issues.apache.org/jira/browse/SPARK-21521
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Adrian Bridgett
>
> (Regression cf. 2.0.2)
> We run spark as several users, these write to the history location where the files are saved as those users with permissions of 770 (this is hardcoded in EventLoggingListener.scala).
> The history service runs as root so that it has permissions on these files (see https://spark.apache.org/docs/latest/security.html).
> This worked fine in v2.0.2, however in v2.2.0 the events are being skipped unless I add the root user into each users group at which point they are seen.
> We currently have all acls configuration unset.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org