You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@zeppelin.apache.org by echarles <gi...@git.apache.org> on 2016/05/30 13:42:25 UTC
[GitHub] incubator-zeppelin pull request: Make path for .spark-dist folder ...
GitHub user echarles opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/930
Make path for .spark-dist folder always accessible
### What is this PR for?
Current Path for .spark-dist is outside the reactor, which can breaks build.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.337 s
[INFO] Finished at: 2016-05-30T15:34:30+02:00
[INFO] Final Memory: 73M/469M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.googlecode.maven-download-plugin:download-maven-plugin:1.2.1:wget (download-pyspark-files) on project zeppelin-spark-dependencies: IO Error: Destination '/opt/zeppelin-master/spark-dependencies/target/../../.spark-dist/spark-1.6.1.tgz' directory cannot be created -> [Help 1]
### What type of PR is it?
[Improvement]
### Todos
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN/ZEPPELIN-902
### How should this be tested?
Build with pyspark profile
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/datalayer/zeppelin-datalayer pyspark-build
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/930.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #930
----
commit cf678ad1ace62c84f0bef23883f1cb591e30e634
Author: Eric Charles <er...@datalayer.io>
Date: 2016-05-30T13:39:25Z
Make path for .spark-dist folder always accessible
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Make path for .spark-dist folder always acc...
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/930
Thanks for explaining!
So as far as I have understood, the pain point is: after `mvn clean`, on the same repo `mvn validate` faild due to `maven-download-plugin` exception
```
iO Error: Destination '/opt/zeppelin-master/spark-dependencies/target/../../.spark-dist/spark-1.6.1.tgz' directory cannot be created
```
who presumably complaining about non-existant `/target/` part of it's destination path?
and what you suggest is that we put there some different path, that is not target-related, and what I suggest is that we better make it still pointing to the same `.spark-dist` dir, one level above the repo root.
`${session.executionRootDirectory}/../.spark-dist` sounds as a good solution to me.
Or another, harder way could be: if you could provide the stack trace of the same error (running `mvn -X ...`) we could look into the implementation of [maven-download-plugin](https://github.com/maven-download-plugin/maven-download-plugin) and see if we could improve it to behave like `mkdir -p` in this case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Make path for .spark-dist folder always acc...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/930
Tried with the ${session.executionRootDirectory} and got [ERROR] around Ant part ...<untar src="${session.executionRootDirectory}/.spark-dist/spark-1.6.1.tgz"
Sounds like maven-antrun-plugin (even last version) is not happy with ${session.executionRootDirectory}
Hacking maven-antrun-plugin or maven-download-plugin is not on my list. I guess we can simply live with this for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Make path for .spark-dist folder ...
Posted by bzz <gi...@git.apache.org>.
Github user bzz commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/930#issuecomment-222575332
Hi, thanks for taking care of the build stability!
`.spark-dist` was introduced as our CI build cache dir, and is used in [multiple places](https://github.com/apache/incubator-zeppelin/search?utf8=%E2%9C%93&q=.spark-dist)
As far as I understand, you suggest using different dirs for bin and src downloads, and src dir will not be part of Travis CI cache any more, so it will be actually dowloaded from Apache mirror on every build again, which is from our previous experience make build very unstable and error-prone.
I do not think it's possible to cache things in the paths, that are going to be overridden by CI's `git clone ...`. Do you have suggestions on how to make it cache friendly and do not break the build?
> .spark-dist is outside the reactor, which can breaks build
Also, could you please help me to understand why the fact of being outside of reactor can break the build?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-zeppelin pull request: Make path for .spark-dist folder always acc...
Posted by echarles <gi...@git.apache.org>.
Github user echarles commented on the pull request:
https://github.com/apache/incubator-zeppelin/pull/930
@bzz Just commented on ZEPPELIN-902 - My concern was on my local build, not on Travis stability.
As mentionned onthe JIRA, the target `target` folder is not present after a clean, because the download-pyspark-files is binded to the `validate` phase.
I guess using ${basedir} could do the job, but then we are not 100% safe because building in the spark-dependencies folder would put the spark-dist at a different location...
Maybe ${session.executionRootDirectory} ? (see http://stackoverflow.com/questions/3084629/finding-the-root-directory-of-a-multi-module-maven-reactor-project).
Another approach would be to bind to the `compile` phase, but I guess it will have side-effects.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---