You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by vanzin <gi...@git.apache.org> on 2014/06/07 01:11:42 UTC
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/1002
Yarn: report HS URL in client mode, correct user in cluster mode.
Yarn client mode was not setting the app's tracking URL to the
History Server's URL when configured by the user. Now client mode
behaves the same as cluster mode.
In SparkContext.scala, the "user.name" system property had precedence
over the SPARK_USER environment variable. This means that SPARK_USER
was never used, since "user.name" is always set by the JVM. In Yarn
cluster mode, this means the application always reported itself as
being run by user "yarn" (or whatever user was running the Yarn NM).
One could argue that the correct fix would be to use UGI.getCurrentUser()
here, but at least for Yarn that will match what SPARK_USER is set
to.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark yarn-client-url
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1002.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1002
----
commit 4c692d96b632488125de7920752c001a1a09eac9
Author: Marcelo Vanzin <va...@cloudera.com>
Date: 2014-06-06T23:06:47Z
Yarn: report HS URL in client mode, correct user in cluster mode.
Yarn client mode was not setting the app's tracking URL to the
History Server's URL when configured by the user. Now client mode
behaves the same as cluster mode.
In SparkContext.scala, the "user.name" system property had precedence
over the SPARK_USER environment variable. This means that SPARK_USER
was never used, since "user.name" is always set by the JVM. In Yarn
cluster mode, this means the application always reported itself as
being run by user "yarn" (or whatever user was running the Yarn NM).
One could argue that the correct fix would be to use UGI.getCurrentUser()
here, but at least for Yarn that will match what SPARK_USER is set
to.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45810108
BTW, just ran into this in SecurityManager.scala:
private val defaultAclUsers = Seq[String](System.getProperty("user.name", ""),
Option(System.getenv("SPARK_USER")).getOrElse(""))
So that probably needs the same treatment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45516754
yarn-alpha seems to already do it:
if (registered) {
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
.asInstanceOf[FinishApplicationMasterRequest]
...
finishReq.setTrackingUrl(sparkConf.get("spark.yarn.historyServer.address", ""))
resourceManager.finishApplicationMaster(finishReq)
}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45948945
Changes look good. Note that this only fixes the url for unsecure YARN cluster. For secure clusters they generally use the web app proxy which requires more setup and will be handled under SPARK-1291.
Thanks @vanzin
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1002
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45926094
(Ah, to clarify, I wasn't looking at the web ui, but at the history server ui. It should sort of show the same info, though, as far as I can tell.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45635778
D'oh. Will update shortly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by jerryshao <gi...@git.apache.org>.
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/1002#discussion_r13683413
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -297,7 +297,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Set SPARK_USER for user who is running SparkContext.
val sparkUser = Option {
- Option(System.getProperty("user.name")).getOrElse(System.getenv("SPARK_USER"))
--- End diff --
Hi @tgravescs , originally I put SPARK_USER before user.name when I submitted a PR, and someone suggested me to change the order to keep consistent with other Spark parameter, so I changed this. I assume it's OK in standalone and mesos mode, but I didn't test it in Yarn mode.
It looks good to me to change this order if needed. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45392998
Note: tested with yarn-client, yarn-cluster and local, checked the spark user listed in the history server for each app.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45500441
Can you also please update for yarn-alpha
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45936763
Ah ok, that makes more sense. The history UI just displays what was recorded as the user on application start event from spark. In the case the daemons are running as user yarn without security it appears this is being put in as user yarn:
{"Event":"SparkListenerApplicationStart","App Name":"JavaWordCount","Timestamp":1402600427658,"User":"yarn"}
I'm a little bit surprised that the change you made fixes that for multiple users. is SPARK_USER set to your user when you start the history server? Does it work with multiple users? We should probably change it to put the correct user in the event log.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45499511
Sorry I misunderstood, this is not SPARK-1291, can you please file jira.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45918304
@tgravescs I have a (non-secured) yarn cluster (where daemons run as "yarn"), and I was launching processes from my own machine (not in the cluster). I'm using the event logger to write app logs to HDFS.
In client mode, my machine is the driver so my user gets correctly written to the app log. But in cluster mode the AM is launched as the same user as the node manager (since it's not a kerberized setup), so the spark user written to the app log is the same user running the node manager, which is "yarn", and not myself.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45950491
Also committed to branch-1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45937328
Ok so looking more at it that is what is happening the postApplicationStart is just using sparkUser and since you changed it, it is properly logging the right thing now. So ignore my questions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45624302
I think that is the ApplicationMaster, I was referring to the ExecutorLauncher for yarn-alpha.
def finishApplicationMaster(status: FinalApplicationStatus) {
logInfo("finish ApplicationMaster with " + status)
val finishReq = Records.newRecord(classOf[FinishApplicationMasterRequest])
.asInstanceOf[FinishApplicationMasterRequest]
finishReq.setAppAttemptId(appAttemptId)
finishReq.setFinishApplicationStatus(status)
resourceManager.finishApplicationMaster(finishReq)
}
We should add a call to finishReq.setTrackingUrl(...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45393038
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/1002#discussion_r13653078
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -297,7 +297,7 @@ class SparkContext(config: SparkConf) extends Logging {
// Set SPARK_USER for user who is running SparkContext.
val sparkUser = Option {
- Option(System.getProperty("user.name")).getOrElse(System.getenv("SPARK_USER"))
--- End diff --
@jerryshao I think you had originally added this logic, Is there a usecase where this order is required? It always seemed odd to me since I expect user.name to always be set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Yarn: report HS URL in client mode, correct us...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45499343
note that this is jira SPARK-1291
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45810178
(Oh wait, that's adding both user.name and SPARK_USER to the default users list? Maybe it works. but looks strange.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45890230
also note that yarn does set the SPARK_USER to UGI in ClientBase
env("SPARK_USER") = UserGroupInformation.getCurrentUser().getShortUserName()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2080] Yarn: report HS URL in client mod...
Posted by tgravescs <gi...@git.apache.org>.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/1002#issuecomment-45889538
Yeah the Securitymanager is adding both to handle the case the containers run as one user but is accessing HDFS as whoever specified in SPARK_USER, it also came up in the review.
@vanzin can you tell me your yarn setup where it was showing yarn as the user in web UI? I have a single node setup with daemons running as yarn and then launch job as myself and it shows up as myself (without your change). Are you running your client as super user and then specifying SPARK_USER to access hdfs files?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---