You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by skonto <gi...@git.apache.org> on 2016/03/22 15:12:17 UTC
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uris to...
GitHub user skonto opened a pull request:
https://github.com/apache/spark/pull/11887
[SPARK-13041][Mesos]add driver sandbox uris to the dispatcher ui
## What changes were proposed in this pull request?
Adds a sandbox link per driver in the dispatcher ui
Implementation would have been straightforward if it was not for this issue:
https://issues.apache.org/jira/browse/MESOS-4992
As a workaround we have to get runtime info from the master and the slave where the driver was launched on as a task.
There is no scalability issues because we use /state-summary instead of state.json
and ask one slave for the directory of the driver's task.
The sandbox uri has the following format:
`http://172.17.0.1:5050/#/slaves/1d01c68e-96e7-4196-a340-4da8e12892c0-S1/browse?path=%2Ftmp%2Fmesos%2Fslaves%2F1d01c68e-96e7-4196-a340-4da8e12892c0-S1%2Fframeworks%2F1d01c68e-96e7-4196-a340-4da8e12892c0-0009%2Fexecutors%2Fdriver-20160322160503-0001%2Fruns%2Fd90b463e-ad52-433e-bcdc-6c3ffbafdc68`
## How was this patch tested?
Tested with the mesos test suite here: https://github.com/typesafehub/mesos-spark-integration-tests.
Attached image shows the ui modification.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/skonto/spark add_sandbox_uri
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11887.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11887
----
commit 83adebb93f270e31a420cca23043806b54fcdf2f
Author: Stavros Kontopoulos <st...@typesafe.com>
Date: 2016-03-22T13:58:05Z
add sandbox to the drivers
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r66418166
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r66418140
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
--- End diff --
yes
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r57069354
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
+ val sandBoxUri = s"$masterUri" +
+ s"/#/slaves/${state.slaveId.getValue}" +
+ s"/browse?path=${directory.get}"
+ <a href={sandBoxUri}>Sandbox</a>
--- End diff --
Yes i know, but had now way to test it... dont really know the correct format there for the url... please provide the specs in our document.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-222111027
Not in 0.29 but soon enough. https://issues.apache.org/jira/browse/MESOS-4992
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200092683
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53826/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/11887
@HyukjinKwon I will close this one and start fresh.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
gentle ping
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r65579412
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
--- End diff --
Kill space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto closed the pull request at:
https://github.com/apache/spark/pull/11887
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/11887
@HyukjinKwon probably. Let me ask Michael. @mgummelt what do you think? what options do we have?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r56994609
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -76,6 +104,7 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
<td>{state.startDate}</td>
<td>{state.slaveId.getValue}</td>
<td>{stateString(state.mesosTaskStatus)}</td>
+ {sandboxCol}
--- End diff --
Not a big deal, but it looks like all the other cells have `<td>` elements in here, and the call just generates the contents of that cell. I'd prefer to keep the same pattern everywhere.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199868189
**[Test build #53780 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53780/consoleFull)** for PR 11887 at commit [`485386d`](https://github.com/apache/spark/commit/485386da81ba07139892f7053ce070267263e337).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/11887
**[Test build #79144 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79144/testReport)** for PR 11887 at commit [`049976f`](https://github.com/apache/spark/commit/049976f67b572fbaacd5cedb532914a024bc8de7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200049448
**[Test build #53826 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53826/consoleFull)** for PR 11887 at commit [`485386d`](https://github.com/apache/spark/commit/485386da81ba07139892f7053ce070267263e337).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r66418116
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
--- End diff --
sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/11887
**[Test build #79144 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79144/testReport)** for PR 11887 at commit [`049976f`](https://github.com/apache/spark/commit/049976f67b572fbaacd5cedb532914a024bc8de7).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200092465
**[Test build #53826 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53826/consoleFull)** for PR 11887 at commit [`485386d`](https://github.com/apache/spark/commit/485386da81ba07139892f7053ce070267263e337).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r57069417
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
--- End diff --
Sounds more ugly than it is im afraid, given the options we have. I have already opened a ticket anyway.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/11887
Great thnx @haosdent :)I will proceed with updating the PR. Thnx a lot!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r57036992
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
+ val sandBoxUri = s"$masterUri" +
+ s"/#/slaves/${state.slaveId.getValue}" +
+ s"/browse?path=${directory.get}"
+ <a href={sandBoxUri}>Sandbox</a>
--- End diff --
This is going to break DCOS and any other system where the dispatcher sits behind a proxy. The master is not directly accessible. You could maybe fix this by adding a proxy prefix property similar to APPLICATION_PROXY_BASE.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r65579276
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
+ val sandBoxUri = s"$masterUri" +
+ s"/#/slaves/${state.slaveId.getValue}" +
+ s"/browse?path=${directory.get}"
+ <a href={sandBoxUri}>Sandbox</a>
--- End diff --
I think we should add a property like @mgummelt suggested to override masterUri if available.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r56999019
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -115,4 +144,58 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
}
sb.toString()
}
+
+ private def getIp4(ip: Int): String = {
+ val buffer = ByteBuffer.allocate(4)
+ buffer.putInt(ip)
+ // we need to check about that because protocolbuf changes the order
+ // which by mesos api is considered to be network order (big endian).
+ val result = if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {
+ buffer.array.toList.reverse
+ } else {
+ buffer.array.toList
+ }
+ result.map{byte => byte & 0xFF}.mkString(".")
+ }
+
+ private def getListFromJson(value: JValue): List[Map[String, Any]] = {
+ value.values.asInstanceOf[List[Map[String, Any]]]
+ }
+
+ private def getTasktDirectory(masterUri: String, driverFwId: String, slaveId: String):
--- End diff --
yes i will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
Hi @skonto, are you able to proceed this PR further? if not, it might be better closed for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by haosdent <gi...@git.apache.org>.
Github user haosdent commented on the issue:
https://github.com/apache/spark/pull/11887
@HyukjinKwon @skonto Sorry for the delay, MESOS-4992 have been fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/11887
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79144/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
gentle ping.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r56994916
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -115,4 +144,58 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
}
sb.toString()
}
+
+ private def getIp4(ip: Int): String = {
+ val buffer = ByteBuffer.allocate(4)
+ buffer.putInt(ip)
+ // we need to check about that because protocolbuf changes the order
+ // which by mesos api is considered to be network order (big endian).
+ val result = if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {
+ buffer.array.toList.reverse
+ } else {
+ buffer.array.toList
+ }
+ result.map{byte => byte & 0xFF}.mkString(".")
+ }
+
+ private def getListFromJson(value: JValue): List[Map[String, Any]] = {
+ value.values.asInstanceOf[List[Map[String, Any]]]
+ }
+
+ private def getTasktDirectory(masterUri: String, driverFwId: String, slaveId: String):
--- End diff --
Probably a typo, this should be `getTaskDirectory`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/11887
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199928789
**[Test build #53780 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53780/consoleFull)** for PR 11887 at commit [`485386d`](https://github.com/apache/spark/commit/485386da81ba07139892f7053ce070267263e337).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r66418456
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
+ val sandBoxUri = s"$masterUri" +
+ s"/#/slaves/${state.slaveId.getValue}" +
+ s"/browse?path=${directory.get}"
+ <a href={sandBoxUri}>Sandbox</a>
--- End diff --
Ok i agree i will look into this one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200027126
Hive tests fail...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r66418176
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -115,4 +142,58 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
}
sb.toString()
}
+
+ private def getIp4(ip: Int): String = {
+ val buffer = ByteBuffer.allocate(4)
+ buffer.putInt(ip)
+ // we need to check about that because protocolbuf changes the order
+ // which by mesos api is considered to be network order (big endian).
+ val result = if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {
+ buffer.array.toList.reverse
+ } else {
+ buffer.array.toList
+ }
+ result.map{byte => byte & 0xFF}.mkString(".")
+ }
+
+ private def getListFromJson(value: JValue): List[Map[String, Any]] = {
+ value.values.asInstanceOf[List[Map[String, Any]]]
+ }
+
+ private def getTaskDirectory(masterUri: String, driverFwId: String, slaveId: String):
+ Option[String] = {
+
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-212461391
This will be fixed in mesos 0.29.0 release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r65579886
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -115,4 +142,58 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
}
sb.toString()
}
+
+ private def getIp4(ip: Int): String = {
+ val buffer = ByteBuffer.allocate(4)
+ buffer.putInt(ip)
+ // we need to check about that because protocolbuf changes the order
+ // which by mesos api is considered to be network order (big endian).
+ val result = if (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN) {
+ buffer.array.toList.reverse
+ } else {
+ buffer.array.toList
+ }
+ result.map{byte => byte & 0xFF}.mkString(".")
+ }
+
+ private def getListFromJson(value: JValue): List[Map[String, Any]] = {
+ value.values.asInstanceOf[List[Map[String, Any]]]
+ }
+
+ private def getTaskDirectory(masterUri: String, driverFwId: String, slaveId: String):
+ Option[String] = {
+
--- End diff --
Remove extra white spaces here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by dragos <gi...@git.apache.org>.
Github user dragos commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199840159
LGTM apart for minor comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r57037689
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
--- End diff --
You're making O(N) HTTP requests. I think that may be an unacceptable performance cost.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199929492
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
@skonto, do you maybe know (or a wild guess) when we would be able to proceed this? Probably, closing this for now and reopening might be an option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200048764
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-212465411
WIP
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r56999047
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -76,6 +104,7 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
<td>{state.startDate}</td>
<td>{state.slaveId.getValue}</td>
<td>{stateString(state.mesosTaskStatus)}</td>
+ {sandboxCol}
--- End diff --
ok i will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r65579463
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
--- End diff --
Remove extra space
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/11887
@HyukjinKwon I will have a look and let you know, please don't close it for now. There was finally progress at the mesos side:
https://reviews.apache.org/r/58872/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by skonto <gi...@git.apache.org>.
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/11887
@tnachen i will fix those. I am also waiting for the fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by mgummelt <gi...@git.apache.org>.
Github user mgummelt commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r57059368
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
--- End diff --
Mesosphere is prioritizing a Mesos fix to make this URL constructable. I propose we wait for that, then maybe do some version checking here to enable this feature.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-200092682
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199873052
**[Test build #53773 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53773/consoleFull)** for PR 11887 at commit [`83adebb`](https://github.com/apache/spark/commit/83adebb93f270e31a420cca23043806b54fcdf2f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199929498
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53780/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199835138
**[Test build #53773 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53773/consoleFull)** for PR 11887 at commit [`83adebb`](https://github.com/apache/spark/commit/83adebb93f270e31a420cca23043806b54fcdf2f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199873306
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53773/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13041][Mesos]add driver sandbox uri to ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11887#issuecomment-199873303
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #11887: [SPARK-13041][Mesos]add driver sandbox uri to the...
Posted by tnachen <gi...@git.apache.org>.
Github user tnachen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11887#discussion_r65579454
--- Diff: core/src/main/scala/org/apache/spark/deploy/mesos/ui/MesosClusterPage.scala ---
@@ -68,6 +75,25 @@ private[mesos] class MesosClusterPage(parent: MesosClusterUI) extends WebUIPage(
private def driverRow(state: MesosClusterSubmissionState): Seq[Node] = {
val id = state.driverDescription.submissionId
+ val masterInfo = parent.scheduler.getSchedulerMasterInfo()
+ val schedulerFwId = parent.scheduler.getSchedulerFrameworkId()
+ val sandboxCol = if (masterInfo.isDefined && schedulerFwId.isDefined) {
+
+ val masterUri = masterInfo.map{info => s"http://${getIp4(info.getIp)}:${info.getPort}"}.get
+ val directory = getTaskDirectory(masterUri, id, state.slaveId.getValue)
+
+ if(directory.isDefined) {
--- End diff --
Space between if (
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
@skonto, Probably, how about addressing the comments above and resolve conflicts for now? I think it is better than leaving this open without any further actions. I have been suggesting to close PRs that are inactive to review comments.
I will ping the guys if it is on a mergeable state. Otherwise, I would rather like to close this for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #11887: [SPARK-13041][Mesos]add driver sandbox uri to the dispat...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/11887
@skonto Sure, I would not propose to close. Thank you for your input.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org