You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by prb <gi...@git.apache.org> on 2015/01/20 21:41:15 UTC
[GitHub] spark pull request: Exclude httpFileServer for local operation.
GitHub user prb opened a pull request:
https://github.com/apache/spark/pull/4125
Exclude httpFileServer for local operation.
This would also make a fine configuration parameter.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/prb/spark local-no-http-file-server
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4125.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4125
----
commit c86026baee389fd44c8cbd7e40c80076110febc9
Author: Paul Brown <pr...@mult.ifario.us>
Date: 2015-01-20T08:05:49Z
Exclude httpFileServer for local operation.
This would also make a fine configuration parameter.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70730880
[Test build #25839 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25839/consoleFull) for PR 4125 at commit [`c86026b`](https://github.com/apache/spark/commit/c86026baee389fd44c8cbd7e40c80076110febc9).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70729892
Jenkins, this is ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70737463
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25839/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by prb <gi...@git.apache.org>.
Github user prb commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70729014
https://issues.apache.org/jira/browse/SPARK-5340
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by prb <gi...@git.apache.org>.
Github user prb closed the pull request at:
https://github.com/apache/spark/pull/4125
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70728973
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70760371
@prb Can you see whether #3130 (shading Jetty in Spark) addresses your use-case?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by prb <gi...@git.apache.org>.
Github user prb commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70744946
I have Spark embedded within a web service that is using Jetty 9, so the classes for Jetty (along with those for Jersey) are unavailable. The system works with this change; I can chase some of the plumbing if there are a known set of touchpoints.
I would argue that the right thing to do is to make all of the non-computation pieces of the system possible to disable conveniently. (This is a selfish wish for people like me who want to use Spark as a library and not a standalone server/service.) That is already the case with the UI and the pretty (but gratuitous) status output.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70737449
[Test build #25839 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25839/consoleFull) for PR 4125 at commit [`c86026b`](https://github.com/apache/spark/commit/c86026baee389fd44c8cbd7e40c80076110febc9).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by prb <gi...@git.apache.org>.
Github user prb commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70945034
Yup. I'll close it up; thanks for the discussion. FWIW, I'd still look at the dependence on `httpFileServer` as a non-feature if not an outright bug. (I can think of lots of reasons (e.g., security) not to want a system to spin-up an HTTP service on a randomly assigned port.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70944319
I don't think that we want to add a mechanism to disable `httpFileServer` since it's necessary for `addFile` / `addJar` to work correctly.
If you'd like to propose new features, could you file issues on the Spark JIRA (https://issues.apache.org/jira/browse/SPARK), since that's the best place to have feature discussions?
Since it looks like #3130 will address this and because we can't easily disable `httpFileServer`, do you mind closing this PR? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70737225
I don't think that you can just null out `httpFileServer` in local mode because methods like `addJar` call methods on it and we have to support those features in local mode (even though they're effectively no-ops) so that jobs designed to run on clusters can be tested locally.
Can you explain your motivation for disabling this in local mode?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Exclude httpFileServer for local operation.
Posted by prb <gi...@git.apache.org>.
Github user prb commented on the pull request:
https://github.com/apache/spark/pull/4125#issuecomment-70765623
The approach in #3130 would probably work for me. As Spark evolves, it would be nice to have the perspective that Spark is a library that other systems (including the various decorations of Spark with UI, etc.) can build on. (I bumped my head on logging initialization issues a while back, too...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org