You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yongtang <gi...@git.apache.org> on 2017/03/08 17:31:22 UTC
[GitHub] spark pull request #17212: [SPARK-17979] Remove deprecated support for confi...
GitHub user yongtang opened a pull request:
https://github.com/apache/spark/pull/17212
[SPARK-17979] Remove deprecated support for config SPARK_YARN_USER_ENV
## What changes were proposed in this pull request?
This fix removes deprecated support for config `SPARK_YARN_USER_ENV`, as is mentioned in SPARK-17979.
## How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yongtang/spark SPARK-17979
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17212.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17212
----
commit 7752fd46da6fe5ca8150cadccb858483a092dd8c
Author: Yong Tang <yo...@outlook.com>
Date: 2017-03-08T17:27:13Z
[SPARK-17979] Remove deprecated support for config SPARK_YARN_USER_ENV
This fix removes deprecated support for config `SPARK_YARN_USER_ENV`,
as is mentioned in SPARK-17979.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17212
And please remove the template text from your PR description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_E...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/17212#discussion_r105467383
--- Diff: docs/hardware-provisioning.md ---
@@ -47,12 +47,6 @@ Storage tab of Spark's monitoring UI (`http://<driver-node>:4040`) to see its si
Note that memory usage is greatly affected by storage level and serialization format -- see
the [tuning guide](tuning.html) for tips on how to reduce it.
-Finally, note that the Java VM does not always behave well with more than 200 GB of RAM. If you
--- End diff --
You had a good point about SPARK_WORKER_INSTANCES being used in standalone mode. I think we might need to keep this doc as a result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17212
I think it's indeed OK to remove at this point because it's not documented, and because of the JIRA discussion. However it's ideal if you can look for other SPARK_* env variables that are long since gone but still checked in the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/17212
This actually covers SPARK-14453 too then. It relates to SPARK-12344 but doesn't fully resolve it. We can link all these in the title though.
Yes, I support removing these props as they have been deprecated a long, long time.
There are more usages of `SPARK_WORKER_INSTANCES` and `SPARK_CLASSPATH` though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_ENV and ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17212
LGTM. I also linked SPARK-17979 to the "parent" of all these cleanup tasks (SPARK-12344); before we completely remove things like `SPARK_WORKER_INSTANCES` we'll need to add proper support for configuration files in the different Spark daemons (master, worker, etc).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_E...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17212
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17212
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by yongtang <gi...@git.apache.org>.
Github user yongtang commented on the issue:
https://github.com/apache/spark/pull/17212
@srowen @vanzin The PR has been updated with the following removed:
```
SPARK_YARN_USER_ENV
SPARK_JAVA_OPTS
SPARK_CLASSPATH
SPARK_WORKER_INSTANCES
```
All of them have already been labeled in the source code:
```
This is deprecated in Spark 1.0+.
```
(The PR description has been updated as well)
Please take a look and let me know if there are any issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated support for config SPARK...
Posted by yongtang <gi...@git.apache.org>.
Github user yongtang commented on the issue:
https://github.com/apache/spark/pull/17212
Thanks @srowen Will take a look at other envs and update the PR if relevant.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_ENV and ...
Posted by yongtang <gi...@git.apache.org>.
Github user yongtang commented on the issue:
https://github.com/apache/spark/pull/17212
Thanks @srowen. The PR has been updated with additional usage of SPARK_CLASSPATH and SPARK_WORKER_INSTANCES removed.
There are `SPARK_WORKER_INSTANCES` used in `start-slave.sh` and `stop-slave.sh` but it looks like they are launching multiple processes so not sure if they should be removed as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_E...
Posted by yongtang <gi...@git.apache.org>.
Github user yongtang commented on a diff in the pull request:
https://github.com/apache/spark/pull/17212#discussion_r105471741
--- Diff: docs/hardware-provisioning.md ---
@@ -47,12 +47,6 @@ Storage tab of Spark's monitoring UI (`http://<driver-node>:4040`) to see its si
Note that memory usage is greatly affected by storage level and serialization format -- see
the [tuning guide](tuning.html) for tips on how to reduce it.
-Finally, note that the Java VM does not always behave well with more than 200 GB of RAM. If you
--- End diff --
@srowen The PR has been updated with the doc kept. Please take a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17212: [SPARK-17979] Remove deprecated SPARK_YARN_USER_ENV and ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/17212
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org