You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by kwmonroe <gi...@git.apache.org> on 2016/11/22 23:46:29 UTC
[GitHub] bigtop pull request #163: BIGTOP-2603: NN/RM charm should include a spark us...
GitHub user kwmonroe opened a pull request:
https://github.com/apache/bigtop/pull/163
BIGTOP-2603: NN/RM charm should include a spark user/group
When NN/RM are not colocated with spark, they do not know about the spark user/group. When in `yarn-client` mode, the spark charm will create hdfs dirs with `spark` ACLs. This leads to errors like this on the namenode:
`WARN org.apache.hadoop.security.UserGroupInformation: No groups available for user spark`
Always create the spark user/group on NN and RM. This also works if spark/NN/RM are all colocated.
Note that `hadoop` should be a secondary group (not primary) for extra users created by these charms, so this PR ensures that as well.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/juju-solutions/bigtop bug/BIGTOP-2603/add-spark-user
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/bigtop/pull/163.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #163
----
commit 74b427eccaafd0fd44f5d5aba2289d166388cdd3
Author: Kevin W Monroe <ke...@canonical.com>
Date: 2016-11-22T23:38:58Z
add spark user/group so they always exist on NN and RM (prevents warnings when NN/RM are not colocated with spark)
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop issue #163: BIGTOP-2603: NN/RM charm should include a spark user/grou...
Posted by ktsakalozos <gi...@git.apache.org>.
Github user ktsakalozos commented on the issue:
https://github.com/apache/bigtop/pull/163
LGTM +1 . Do you think we should be doing the same with other components that may use hdfs, like hbase?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop issue #163: BIGTOP-2603: NN/RM charm should include a spark user/grou...
Posted by kwmonroe <gi...@git.apache.org>.
Github user kwmonroe commented on the issue:
https://github.com/apache/bigtop/pull/163
Yeah, we probably should. `oozie` and `hue` come to mind. We chatted about this a little while ago. It would really be nice if the charm that relied on `hadoop-plugin` would send user info over the plugin relation.
The plugin would then proxy that to the NN or RM as needed. Unfortunately, this isn't really a priority at the moment, so I think we'll treat `spark` as special for now and work up a more generic solution later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] bigtop pull request #163: BIGTOP-2603: NN/RM charm should include a spark us...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/bigtop/pull/163
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---