You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by loachli <gi...@git.apache.org> on 2014/08/23 05:42:55 UTC
[GitHub] spark pull request: Update building-with-maven.md
GitHub user loachli opened a pull request:
https://github.com/apache/spark/pull/2102
Update building-with-maven.md
Support building spark under http proxy environment
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/loachli/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2102
----
commit 1f1d55172bbe4dc3fc3bef310ff86a679e321970
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-23T03:04:41Z
Update building-with-maven.md
Support building spark under http proxy environment
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/2102#discussion_r16629723
--- Diff: docs/building-with-maven.md ---
@@ -156,4 +156,12 @@ then ship it over to the cluster. We are investigating the exact cause for this.
The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
+# Building under http proxy environment
+Sometimes,spark need be built in http proxy environment, We recommend the following settings:
+
+ mvn -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -DskipTests clean package
--- End diff --
I'm not sure this resolves proxy problems. It may resolve a particular issue wherein your network proxy is breaking SSL connections, but is not how you configure proxies. See http://maven.apache.org/guides/mini/guide-proxies.html
Attacks are very rare, but if someone were trying to inject a bad binary into your build, this would invite users to explicitly ignore that warning sign. So I disagree that this is something all users should use when using a proxy.
The `yarn` profile and so on are not related, just the two `maven.wagon` settings. The error message you quote does not contain the type of failure you would see, which is a "could not resolve dependencies" error. Finally, there are punctuation and capitalization problems, like "http".
I don't think this should be added.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-54056189
I have created SPARK-3191(https://issues.apache.org/jira/browse/SPARK-3191) for it. Do you think it is enough for this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-62348129
Hi there - I thought a bit more about this and I think we probably shouldn't explicitly tell users to disable security settings. I've never heard of a user report this issue before - so it doesn't seem super common, even for users building behind proxies (which I know many people do). However, let's do this. If we have other users report this issue again, we can add something to the docs and we'll use this patch as a starting point. So let's close this issue for now and re-open it later if necessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53170070
hey @pwendell , thanks for your comments- Yes, my proxy support https.
I had used no-proxy open environment before. In order to support more people using spark, I have to move spark environment into my company's inner environment. For security reason, I have to use http-proxy provided by my company to access network.
When I used spark in my company's inner environment, I could not compile spark successfully.
Because the maven's error hit was not obvious, I spent much time solving this problem.
You can find the definition of these two parameters in http://maven.apache.org/wagon/wagon-providers/wagon-http/
"maven.wagon.http.ssl.insecure = true/false (default false), enable/disable use of relaxed ssl check for user generated certificates.
maven.wagon.http.ssl.allowall = true/false (default false), enable/disable match of the server's X.509 certificate with hostname. If disabled, a browser like check will be used."
I also found someone else had met this issue (https://issues.apache.org/jira/browse/SPARK-1125)
So I believe, this issue will be met by others in the future.
I still think we could add this hit to the document. One optional way is that I could add risk warnig when using these parameters. Do you agree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53668235
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53170332
Hey @srowen, thanks a lot for your detailed comment.
this is my configuration
<proxy>
<id>lzb</id>
<active>true</active>
<protocol>http</protocol>
<username>user</username>
<password>password</password>
<host>proxy.company.com</host>
<port>8080</port>
<nonProxyHosts>*.company.com</nonProxyHosts>
</proxy>
This configuration is same as the content in the link provided by you
I agree with you that the attack is very rare, especially when I use pom.xml provided by spark
If my suggestion could be accepted, I can reorganized my documents to meet requirements
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli closed the pull request at:
https://github.com/apache/spark/pull/2102
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53668256
I have reorganized my documents to meet requriments. If you don't set this, make-distribution.sh cann't be run properly too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
GitHub user loachli reopened a pull request:
https://github.com/apache/spark/pull/2102
Update building-with-maven.md
Support building spark under http proxy environment
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/loachli/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2102.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2102
----
commit 1f1d55172bbe4dc3fc3bef310ff86a679e321970
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-23T03:04:41Z
Update building-with-maven.md
Support building spark under http proxy environment
commit c41618510e8a6a42265b7591b890530cd0913a56
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-28T02:48:44Z
Update building-with-maven.md
commit 51fd9ef869fab36772423a690079a23242f24327
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-28T02:50:54Z
Update building-with-maven.md
commit 53a8caf18d7b578827407f96dbe45f8f699fe0d7
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-28T02:53:23Z
Update building-with-maven.md
commit 1aa816ba119bbf56bd6c6b21f9ba1ad2db5d8545
Author: lizhengbing <zh...@huawei.com>
Date: 2014-08-28T02:55:25Z
Update building-with-maven.md
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53155103
Hey @loachli - thanks for looking into this. I don't think we can advise users to disable security settings for their maven build. Does your proxy support HTTPS?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-54694402
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by loachli <gi...@git.apache.org>.
Github user loachli commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53141872
I have create SPARK-3191(https://issues.apache.org/jira/browse/SPARK-3191) to describe this issues detailly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2102
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53977986
Can you create a JIRA for this issue? I'm going to re-word this a bit when I merge it, I think it's fine to say that certain users have reported issues building behind proxies without disabling security - and have a clear warning that this, indeed, disables security. Anyways I can fix-up the wording if you create a JIRA for it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Update building-with-maven.md
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2102#issuecomment-53141658
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org