You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by luchesar <gi...@git.apache.org> on 2014/10/21 23:10:21 UTC
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
GitHub user luchesar opened a pull request:
https://github.com/apache/spark/pull/2883
Sbt and Maven builds pass on Linux boxes with encrypted folder
The contribution is my original work. I license the work to the project under the project's open source license.
Building on a Linux box with encrypted home folder results in a failure.
```
[error] == Expanded type of tree ==
[error]
[error] ConstantType(value = Constant(Throwable))
[error]
[error] uncaught exception during compilation: java.io.IOException
[error] File name too long
[error] two errors found
```
This pull request fixes the problem in both the sbt and maven builds.
Few other people are experiencing the same problem and will be good to be fixed.
http://apache-spark-user-list.1001560.n3.nabble.com/compiling-spark-source-code-td13980.html
http://mail-archives.apache.org/mod_mbox/spark-user/201312.mbox/%3C35F52C98-1178-480C-8D84-619D8F341D3E@gmail.com%3E
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/luchesar/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2883.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2883
----
commit 603456a0cb483612df6671e06367726bec8d454b
Author: Luchesar Cekov <lu...@gmail.com>
Date: 2014-10-21T20:44:29Z
SBT build compiles without errors on linux with encrypted home folder
commit 4316adda85695baabe5496e4cead345ae3598b5a
Author: Luchesar Cekov <lu...@gmail.com>
Date: 2014-10-21T21:04:01Z
Maven build compiles without errors on linux with encrypted home folder
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-66534056
Hey Luchesar, thanks for reporting this and providing a workaround. I've created the JIRA here:
https://issues.apache.org/jira/browse/SPARK-4820
I'd like to see how many other users hit this and if we find it is a common issue we can merge it. For now let's close this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-59999593
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by luchesar <gi...@git.apache.org>.
Github user luchesar commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60047780
I have a Linux box with encrypted home folder on ext4 File system. Ext4's max file name length is 255 bytes.
scalac will generate class file names like that for closures.
CurrencyInitializer$$anonfun$com$gottex$gottware$server$startup$initializers$impl$currency$CurrencyInitializer$$updateDepositEquivalentBonds$1.class
On encrypted files system the file names get encrypted and get much longer then usual. This leads to failing build for me on my my machine and a quick search showed other people are experiencing the same problem.
Setting -Xmax-classfile-name scalac parameter tells scalac to generate class file names with length at max X bytes. I find 128 works very well for all the projects I've worked for.
This is nice to have but not mandatory really, I just need to set these every time I build Spark on my local machine plus I can build it on unencrypted partition if I had one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60010425
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22008/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-66534936
Patrick, anyone who tries to build Spark on an encrypted Linux fs will hit it. It's not something that happens for some and not for others.
Fortunately there's an easy workaround (build on a non-encrypted directory). I'm not sure whether that scalac option has any side effects other than just changing generated class names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/2883
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by vanzin <gi...@git.apache.org>.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-66535273
(BTW, when I researched this a long ago, I remember seeing that the max allowed file name for ecryptfs is around 130 bytes or thereabouts.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by medale <gi...@git.apache.org>.
Github user medale commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60467372
I ran into the exact same "file name too long" problem when running the Maven build on an encrypted ext4 partition. The changes in the pom.xml file in this pull request solved the problem and it now builds fine. Thank you for sharing!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60013957
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by luchesar <gi...@git.apache.org>.
Github user luchesar commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60065574
Do you mean API interoperability or just random problems? I guess the generated class names are purely used by the compiler and are not related to any public classes so API interoperability should be fine.
I do use similar setup with Intellij Idea 13 and sbt on a daily basis without any problems.
The value of 128 is of course an engineering guess. I haven't done any research to how long is the encrypted version of a 128 byte file name on ext4 or other file systems. There is a potential this will be different for different file systems e.g OSX's HFS. I guess on HFS and NTFS the encryption doesn't change the file name length as they have 255 and 256 max file length as well and the build is fine on those encrypted file systems.
I can guarantee it works for encrypted ext4 though.
I wanted to create this pull request partly to highlight the problem and to offer a solution for other people who can be baffled with Spark failing to build what looked like a build problem in the beginning.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60010418
[QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22008/consoleFull) for PR 2883 at commit [`4316add`](https://github.com/apache/spark/commit/4316adda85695baabe5496e4cead345ae3598b5a).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60048089
I wonder if this has any other implications for the build though, as class names generated by the build could be different from those generated by, say, compiling with a different tool or an IDE. Compared to the value of supporting this particular use case of developing on a particular encrypted file system.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by aarondav <gi...@git.apache.org>.
Github user aarondav commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60000426
Jenkins, ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-66531140
Yeah since there are some potential downstream interactions with this, I'm going to create a JIRA to document clearly the issue and provide this work-around for users. If enough other people run into this problem we can merge the fix into Spark and experiment with it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60001367
[QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22008/consoleFull) for PR 2883 at commit [`4316add`](https://github.com/apache/spark/commit/4316adda85695baabe5496e4cead345ae3598b5a).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Sbt and Maven builds pass on Linux boxes with ...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/2883#issuecomment-60013457
I'm curious what the problem was and why this fixes it? It compiles fine already on Linux so what does encryption have to do with it? is it likely to affect anyone else's build?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org