You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by tzolov <gi...@git.apache.org> on 2014/09/21 12:22:05 UTC

[GitHub] spark pull request: [Build] SPARK-3624: Failed to find Spark assem...

GitHub user tzolov opened a pull request:

    https://github.com/apache/spark/pull/2477

    [Build] SPARK-3624: Failed to find Spark assembly in /usr/share/spark/lib...

    Define a 'lib' symlink like this:
    
        lib -> /usr/share/spark/jars
    
    This required jdeb maven plugin update from version 0.11 to 1.3

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tzolov/spark SPARK-3624

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/2477.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2477
    
----
commit 24a9bd5b51c26bf2225be7d898bba2c17dc4b15c
Author: tzolov <ch...@gmail.com>
Date:   2014-09-21T10:11:59Z

    [Build] SPARK-3624:Failed to find Spark assembly in /usr/share/spark/lib for RELEASED debian packages

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request: [Build] SPARK-3624: Failed to find Spark assem...

Posted by markgrover <gi...@git.apache.org>.

Github user markgrover commented on the pull request:

https://github.com/apache/spark/pull/2477#issuecomment-58454665

Thanks for including me, @JoshRosen!

I agree with @markhamstra here.

If I *had* to make this work, I wouldn't do the symlink. In fact, I'd update compute-classpath.sh, to, in case of a RELEASE, look under $FWDIR/lib AND $FWDIR/jars for the assembly jar. Symlink may cause some pain down the road, so I'd prefer a backwards compatible and simple change in the compute-classpath script.
I am not an expert on how the jdeb plugin is being used in Spark, but I downloaded the latest binary tarball of spark and there is a "lib" directory there. I can see jdeb taking the contents of lib directory and essentially putting them under jars directory but I think it's a slippery slope to create the symlink. If somehow a lib directory shows up under /usr/share/spark/, this symlink creation would fail.

However, I also want to take this opportunity to raise a meta point. Hadoop, and other projects when they were just starting, took the route of bundling their own rpm/deb packaging code along with the project. They soon realized that it was very hard to maintain that since the bread and butter of the project is writing good, solid distributed software and not figuring out why /var/run is a tempfs on latest Operating Systems. This is where Apache Bigtop was born out of - to house the packaging for all projects of Hadoop ecosystem and to integrate them with each other. Consequently, Apache Bigtop packaging is considered the de-facto packaging for Hadoop ecosystem projects. And, Spark is a part of it, has been for quite a while. So, while it's your call at the end of the day, I'd strongly encourage you to separate the packaging concern to Apache Bigtop since I think it's unreasonable to expect a development project to accommodate the needs of all Operating Systems and packaging stan
dards.

In interest of full-disclosure, I am a committer on the Apache Bigtop project.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org