You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/29 07:10:28 UTC
[GitHub] [spark-website] yutoacts opened a new pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
yutoacts opened a new pull request #350:
URL: https://github.com/apache/spark-website/pull/350
Document "local-cluster" mode on developer-tools.md.
Related links:
https://github.com/apache/spark/pull/33568
https://github.com/apache/spark/pull/33537
https://issues.apache.org/jira/browse/SPARK-36335
https://issues.apache.org/jira/browse/SPARK-595
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts commented on a change in pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts commented on a change in pull request #350:
URL: https://github.com/apache/spark-website/pull/350#discussion_r679593398
##########
File path: site/sitemap.xml
##########
@@ -907,10 +907,6 @@
<loc>https://spark.apache.org/mllib/</loc>
<changefreq>weekly</changefreq>
</url>
-<url>
Review comment:
Reverted. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts edited a comment on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts edited a comment on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889578852
Thanks for the suggestions. If it goes in main docs, should it be documented in `docs/spark-standalone.md` or `docs/submitting-applications.md` (as my initial PR: https://github.com/apache/spark/pull/33537)? IMO documenting it in `docs/spark-standalone.md` might confusing people with standalone mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] tgravescs commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889172500
so I guess it isn't that big of deal because we don't do document it now and doesn't seem to have been a big problem, but I assume this issue was filed for a reason and my concern is that people know about and use local-cluster mode so why not just clarify what its for. there are 2 things, one is obscure it by not documenting it which works for some people, but it doesn't work for others that know about it or find it but don't know its for unit testing only. If we document all the run modes in a common places seems like it would be easier to find for users.
While it does run like standalone mode it doesn't make sense to me to put under standalone mode docs, at least I wouldn't think to go there to look for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889154172
I see, if this is really intended as a developer tool, this would be the right place. The very old SPARK-595 thread suggests it isn't totally for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889188101
My only concern about documenting in the main docs is that it happens to force us to investigate/document it together whenever a cluster related feature (like archive, resource profile, etc.), and then it gives the dev some more overhead of investigation, for example, the one instance in the main document (https://spark.apache.org/docs/latest/configuration.html#custom-resource-scheduling-and-configuration-overview).
If we'll document once, and explicitly say there's no gurnatee on such features since that's a test only mode, I'm fine with doing it in the main docs too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889574165
I am fine either way, no big deal. I will defer to @srowen and @tgravescs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts edited a comment on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts edited a comment on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889578852
Thanks for the suggestions. If it goes in main docs, should it be documented in `docs/spark-standalone.md` or `docs/submitting-applications.md` (as my initial PR: https://github.com/apache/spark/pull/33537)? IMHO documenting it in `docs/spark-standalone.md` might confusing people with standalone mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889578852
Thanks for the suggestions. If it goes in main docs, should it be documented in `docs/spark-standalone.md` or `docs/submitting-applications.md` (as my initial PR: https://github.com/apache/spark/pull/33537)? IMO documenting in `docs/spark-standalone.md` might confusing people with standalone mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts commented on a change in pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts commented on a change in pull request #350:
URL: https://github.com/apache/spark-website/pull/350#discussion_r680600573
##########
File path: site/developer-tools.html
##########
@@ -793,6 +793,18 @@ <h4>In Spark unit tests</h4>
The platform-specific paths to the profiler agents are listed in the
<a href="http://www.yourkit.com/docs/80/help/agent.jsp">YourKit documentation</a>.</p>
+<p><a name="local-cluster"></a></p>
+<h3>Local-cluster mode</h3>
+
+<p>When launching applications with spark-submit, besides options in
+<a href="https://spark.apache.org/docs/latest/submitting-applications.html#master-urls">Master URLs</a>
+, set local-cluster option to emulate a distributed cluster in a single JVM.</p>
Review comment:
I thought it is not necessary as it's on a developer-tools page but I'm totally fine with explicitly saying that here.
BTW if local-cluster mode ends up written in main documents (https://github.com/apache/spark/pull/33537), should it still be documented here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] srowen commented on a change in pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #350:
URL: https://github.com/apache/spark-website/pull/350#discussion_r679162872
##########
File path: site/sitemap.xml
##########
@@ -907,10 +907,6 @@
<loc>https://spark.apache.org/mllib/</loc>
<changefreq>weekly</changefreq>
</url>
-<url>
Review comment:
(Revert this change)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] HyukjinKwon commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889145171
I actually suggested to avoid documenting it in the main docs because local-cluster is test-only mode. But I am fine with doing it in `docs/spark-standalone.md` too. Looks like Tom wants to have it in main docs too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889139082
This doesn't quite feel like the right place to document this. How about docs/spark-standalone.md in the main Spark project docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] tgravescs commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
tgravescs commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-889173380
Note if others disagree, I'm fine with leaving here in developer docs. I would rather see it go in common docs where we describe all run modes or in developer docs to obscure from users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] tgravescs commented on a change in pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
tgravescs commented on a change in pull request #350:
URL: https://github.com/apache/spark-website/pull/350#discussion_r679883614
##########
File path: site/developer-tools.html
##########
@@ -793,6 +793,18 @@ <h4>In Spark unit tests</h4>
The platform-specific paths to the profiler agents are listed in the
<a href="http://www.yourkit.com/docs/80/help/agent.jsp">YourKit documentation</a>.</p>
+<p><a name="local-cluster"></a></p>
+<h3>Local-cluster mode</h3>
+
+<p>When launching applications with spark-submit, besides options in
+<a href="https://spark.apache.org/docs/latest/submitting-applications.html#master-urls">Master URLs</a>
+, set local-cluster option to emulate a distributed cluster in a single JVM.</p>
Review comment:
shouldn't we explicitly say this is for unit testing only?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts commented on pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts commented on pull request #350:
URL: https://github.com/apache/spark-website/pull/350#issuecomment-901576855
It ended up as https://github.com/apache/spark/pull/33537.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org
[GitHub] [spark-website] yutoacts closed pull request #350: [SPARK-36335] Add local-cluster docs to developer-tools.md
Posted by GitBox <gi...@apache.org>.
yutoacts closed pull request #350:
URL: https://github.com/apache/spark-website/pull/350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org