You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by marmbrus <gi...@git.apache.org> on 2015/04/17 00:40:10 UTC
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
GitHub user marmbrus opened a pull request:
https://github.com/apache/spark/pull/5546
[BUILD] Support building with SBT on encrypted Linux partitions
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/marmbrus/spark buildLinux
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5546.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5546
----
commit cc3bd0628200da532d03fb17666456e1aa278142
Author: Michael Armbrust <mi...@databricks.com>
Date: 2015-04-16T22:39:41Z
[BUILD] Support building with SBT on encrypted Linux partitions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93887188
[Test build #30452 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30452/consoleFull) for PR 5546 at commit [`031c602`](https://github.com/apache/spark/commit/031c6025113c064b6fc0b5895b1830f223f6cf55).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93877119
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93873670
[Test build #30447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30447/consoleFull) for PR 5546 at commit [`031c602`](https://github.com/apache/spark/commit/031c6025113c064b6fc0b5895b1830f223f6cf55).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-94599321
You are right that it will effect how the class is found, but if I understand correctly, we are shortening generated classes that will only be referenced internally. For example, a class that is created for a closure inside of a function such as `StreamingJobProgressListener$$anonfun$lastReceivedBatchRecords$1$$anonfun$apply$5$$anonfun$apply$6.class`
I think building on encrypted file systems is becoming more common. Most corporate laptops I have owned required it, and it was the default when I installed ubuntu on my desktop.
Regarding doing it for maven, we can. I was going going for the minimal change that would make my dev workflow easier.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-94884549
Yeah it's becoming much more common so I'd say let's just go for it across both builds. Otherwise if there is some subtle issue caused by this, it will be super confusing that it only affects one build too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93855845
As I recall from previous discussions, this was lightly rejected on the grounds that it created artifacts that are incompatible in subtle ways since the classes are named differently, and that might be too confusing:
https://issues.apache.org/jira/browse/SPARK-4820
Hm, I thought it was someone else besides just me saying that ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93873674
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30447/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93877539
[Test build #30452 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30452/consoleFull) for PR 5546 at commit [`031c602`](https://github.com/apache/spark/commit/031c6025113c064b6fc0b5895b1830f223f6cf55).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93872501
Talk with @pwendell off-line and seems since we don't publish with SBT this is pretty safe. He asked me to update the docs to make it clear why we don't do this for maven (though if someone from the scala side says this is safe, I'd argue to do it there too).
@srowen any further objections to merging this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93937847
Hm, doesn't the class file name affect how it's found? that's how the classloader finds the class. I also don't know of a specific instance where this created a problem, but the fact that it needs to be set means something about the output will change. What was the additional bit of info you mention that says this is safe? If it really is verifiably never going to change the linking result, then it doesn't matter, but seems like it would by changing file names.
Is building on an encrypted file system common? Yes the SBT build would only be for developers though the inconsistency is a little worrying, so I think it best to do it both places or neither.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93866034
[Test build #30441 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30441/consoleFull) for PR 5546 at commit [`cc3bd06`](https://github.com/apache/spark/commit/cc3bd0628200da532d03fb17666456e1aa278142).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-94728926
Yeah I doubt it will affect any user-facing code since those class names aren't massive. It might affect compatibility between two builds of Spark, since these functions get referenced in closures. It might be surprising that Spark built at the same tag on encrypted vs unencrypted file systems isn't compatible. That said I don't have any concrete evidence of this being a problem in practice. I think this is fine as a 'canary'? Nit: maven -> Maven in the doc change. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93855690
[Test build #30441 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30441/consoleFull) for PR 5546 at commit [`cc3bd06`](https://github.com/apache/spark/commit/cc3bd0628200da532d03fb17666456e1aa278142).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93866044
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30441/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93857336
No there are no known issues, just general wondering whether this could mess up compatibility in some subtle way, for instance if some longer exposed classes are renamed because of this. But I guess it only changes the filename and not the bytecode name of the class, so it should be fine.
Michael - maybe we should also update it for maven now as well if we're going to do this (see the JIRA) and close the JIRA. Otherwise, pretty likely someone will post the same thing for Maven.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93887194
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30452/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by marmbrus <gi...@git.apache.org>.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93856573
Ah, thanks for the context @srowen!
@pwendell, do we have any concrete reports of this actually causing any incompatibilities in practice? Given that the change is only going to affect the SBT build (which is never used for distribution artifacts), I'm inclined to merge this patch to simplify the development workflow. As an anecdote, we have been using this option in our internal build (of Spark and other code) for over a year now and have not encountered any problems.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [BUILD] Support building with SBT on encrypted...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5546#issuecomment-93872609
[Test build #30447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/30447/consoleFull) for PR 5546 at commit [`031c602`](https://github.com/apache/spark/commit/031c6025113c064b6fc0b5895b1830f223f6cf55).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org