You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sarutak <gi...@git.apache.org> on 2015/04/09 09:29:27 UTC
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
GitHub user sarutak opened a pull request:
https://github.com/apache/spark/pull/5437
[SPARK-2673][CORE][WEBUI] Enable to attach Debugger to Executors easily
In current implementation, it's difficult to attach debugger to each Executor in the cluster.
There are some reasons as follows.
1) Multi Executors can run on the same machine so each executor should open individual debug ports.
2) Even if we can open unique debug port to each Executors running on the same machine, it's a bother to check debug port of each executor.
To solve those problem, I think following 2 improvement is needed.
1) Enable executor to open unique debug port on a machine by setting debug port to 0.
2) Expand WebUI to be able to show debug ports opening in each executor.
Following image is how Web UI(ExecutorsPage) show the debug ports for each Executor.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sarutak/spark debug-port-feature
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5437.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5437
----
commit a2d75722a13088d94c6cc986d753d944adf7ab10
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-06T11:29:45Z
Initial commit
commit 5c4982c9413c9ed6985e8e8f614730c5cd2c1023
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-07T09:08:41Z
Improved code
commit e87175319c0eedb3d48b825db9d8628946e3e303
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-08T01:05:12Z
Merge branch 'master' of git://git.apache.org/spark into debug-port-feature
commit 602c335f7c382b1db78549386216adf2307ee040
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-08T08:27:49Z
Added a test case to YarnClusterSuite for the debug port feature
commit 4c16e9cb4f0d84927494279650b5b75295f61478
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-08T09:05:54Z
Temporary commit
commit 654500d89ffdb94f9e2c83bc18c7e92793fb524b
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-09T07:11:11Z
Added test cases to UISeleniumSuite
commit dbf69f39b0443dc7eff49cf985b2ac9cc658c954
Author: Kousuke Saruta <sa...@oss.nttdata.co.jp>
Date: 2015-04-09T07:11:36Z
Added ExecutorDebugPortsuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91172032
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29932/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by WangTaoTheTonic <gi...@git.apache.org>.
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91248495
What about ApplicationMaster (ExecutorLauncher) and Driver side? As there may be too much executors so we attach debuggers to them for convienience, and there's no same trouble in AM and Driver, am I right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91137764
[Test build #29929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29929/consoleFull) for PR 5437 at commit [`dbf69f3`](https://github.com/apache/spark/commit/dbf69f39b0443dc7eff49cf985b2ac9cc658c954).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91175081
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29933/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91175070
[Test build #29933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29933/consoleFull) for PR 5437 at commit [`e3ba120`](https://github.com/apache/spark/commit/e3ba120e661735909c3ded2130dcb85d7598cf75).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/5437#discussion_r28051919
--- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ---
@@ -20,6 +20,8 @@ package org.apache.spark.executor
import java.net.URL
import java.nio.ByteBuffer
+import sun.misc.VMSupport
--- End diff --
This is risky to use, even if it is in OpenJDK and Oracle's JVM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91172018
[Test build #29932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29932/consoleFull) for PR 5437 at commit [`79ee2f9`](https://github.com/apache/spark/commit/79ee2f9e77e0dd7ec7da1c692c2d283f5794e17f).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91146093
[Test build #29933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29933/consoleFull) for PR 5437 at commit [`e3ba120`](https://github.com/apache/spark/commit/e3ba120e661735909c3ded2130dcb85d7598cf75).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91192902
Referring to the JIRA, I do wonder how much this adds over just setting up the agent manually with JVM opts, because it touches a reasonable amount of code. Yes, you have to pick a fixed port, but then problem 2 goes away; you know what the agent port is. Yes, that doesn't work with several executors per machine, but if you're restarting your app anyway to debug, would you not perhaps just run a leaner config with 1 per machine? in the event that you really have to debug a multiple executor per machine job, maybe you can make the port a function of the executor's ID? I think we added some support for that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91143778
[Test build #29932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29932/consoleFull) for PR 5437 at commit [`79ee2f9`](https://github.com/apache/spark/commit/79ee2f9e77e0dd7ec7da1c692c2d283f5794e17f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91137619
[Test build #29929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29929/consoleFull) for PR 5437 at commit [`dbf69f3`](https://github.com/apache/spark/commit/dbf69f39b0443dc7eff49cf985b2ac9cc658c954).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-136944437
Thanks for the notification. Unfortunately, now I don't have time to improve this patch. So I'll close for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak closed the pull request at:
https://github.com/apache/spark/pull/5437
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by sarutak <gi...@git.apache.org>.
Github user sarutak commented on a diff in the pull request:
https://github.com/apache/spark/pull/5437#discussion_r28052012
--- Diff: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ---
@@ -20,6 +20,8 @@ package org.apache.spark.executor
import java.net.URL
import java.nio.ByteBuffer
+import sun.misc.VMSupport
--- End diff --
So, what about enabling when we build with a specific profile?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by andrewor14 <gi...@git.apache.org>.
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-136918531
@sarutak are you still actively working on this? If not, would you mind closing this patch for now? If there's interest in the future we can always reopen it against the latest master branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-2673][CORE][WEBUI] Enable to attach Deb...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5437#issuecomment-91137770
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29929/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org