You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mayur Rustagi <ma...@gmail.com> on 2014/07/28 03:32:44 UTC

Spark as a application library vs infra

Based on some discussions with my application users, I have been trying to come up with a standard way to deploy applications built on Spark

1. Bundle the version of spark with your application and ask users store it in hdfs before referring it in yarn to boot your application
2. Provide ways to manage dependency in your app across various versions of spark bundled in with Hadoop distributions 

1 provides greater control and reliability as I am only working against yarn versions and dependencies, I assume 2 gives me some benefits of distribution versions of spark (easier management, common sysops tools ?? ) . 
I was wondering if anyone has thoughts around both and any reasons to prefer one over the other. 

Sent from my iPad

Re: Spark as a application library vs infra

Posted by Sandy Ryza <sa...@cloudera.com>.
At Cloudera we recommend bundling your application separately from the
Spark libraries.  The two biggest reasons are:
* No need to modify your application jar when upgrading or applying a patch.
* When running on YARN, the Spark jar can be cached as a YARN local
resource, meaning it doesn't need to be transferred every time.



On Sun, Jul 27, 2014 at 8:52 PM, Tobias Pfeiffer <tg...@preferred.jp> wrote:

> Mayur,
>
> I don't know if I exactly understand the context of what you are asking,
> but let me just mention issues I had with deploying.
>
> * As my application is a streaming application, it doesn't read any files
> from disk, so therefore I have no Hadoop/HDFS in place and I there is no
> need for it, either. There should be no dependency on Hadoop or HDFS, since
> you can perfectly run Spark applications without it.
> * I use Mesos and so far I always had the downloaded Spark distribution
> accessible for all machines (e.g., via HTTP) and then added my application
> code by uploading a jar built with `sbt assembly`. As the Spark code itself
> must not be contained in that jar file, I had to add '% "provided"' in the
> sbt file, which in turn prevented me from running the application locally
> from IntelliJ IDEA (it would not find the libraries marked with
> "provided"), I always had to use `sbt run`.
> * When using Mesos, on the Spark slaves the Spark jar is loaded before the
> application jar, and so the log4j file from the Spark jar is used instead
> of my custom one (that is different when running locally), so I had to edit
> that file in the Spark distribution jar to customize logging of my Spark
> nodes.
>
> I wonder if the two latter problems would vanish if the Spark libraries
> were bundled together with the application. (That would be your approach
> #1, I guess.)
>
> Tobias
>

Re: Spark as a application library vs infra

Posted by Tobias Pfeiffer <tg...@preferred.jp>.
Mayur,

I don't know if I exactly understand the context of what you are asking,
but let me just mention issues I had with deploying.

* As my application is a streaming application, it doesn't read any files
from disk, so therefore I have no Hadoop/HDFS in place and I there is no
need for it, either. There should be no dependency on Hadoop or HDFS, since
you can perfectly run Spark applications without it.
* I use Mesos and so far I always had the downloaded Spark distribution
accessible for all machines (e.g., via HTTP) and then added my application
code by uploading a jar built with `sbt assembly`. As the Spark code itself
must not be contained in that jar file, I had to add '% "provided"' in the
sbt file, which in turn prevented me from running the application locally
from IntelliJ IDEA (it would not find the libraries marked with
"provided"), I always had to use `sbt run`.
* When using Mesos, on the Spark slaves the Spark jar is loaded before the
application jar, and so the log4j file from the Spark jar is used instead
of my custom one (that is different when running locally), so I had to edit
that file in the Spark distribution jar to customize logging of my Spark
nodes.

I wonder if the two latter problems would vanish if the Spark libraries
were bundled together with the application. (That would be your approach
#1, I guess.)

Tobias

Re: Spark as a application library vs infra

Posted by Krishna Sankar <ks...@gmail.com>.
   - IMHO, #2 is preferred as it could work in any environment (Mesos,
   Standalone et al). While Spark needs HDFS (for any decent distributed
   system) YARN is not required at all - Meson is a lot better.
   - Also managing the app with appropriate bootstrap/deployment framework
   is more flexible across multiple scenarios, topologies et al.
   - What kind of capabilities are you thinking of ? Automatic discovery ?
   Dynamic deployment based on resources available, versions of Hadoop et al ?

Cheers
<k/>



On Sun, Jul 27, 2014 at 6:32 PM, Mayur Rustagi <ma...@gmail.com>
wrote:

> Based on some discussions with my application users, I have been trying to
> come up with a standard way to deploy applications built on Spark
>
> 1. Bundle the version of spark with your application and ask users store
> it in hdfs before referring it in yarn to boot your application
> 2. Provide ways to manage dependency in your app across various versions
> of spark bundled in with Hadoop distributions
>
> 1 provides greater control and reliability as I am only working against
> yarn versions and dependencies, I assume 2 gives me some benefits of
> distribution versions of spark (easier management, common sysops tools ?? )
> .
> I was wondering if anyone has thoughts around both and any reasons to
> prefer one over the other.
>
> Sent from my iPad

Re: Spark as a application library vs infra

Posted by Koert Kuipers <ko...@tresata.com>.
i used to do 1) but couldnt get it to work on yarn and the trend seemed
towards 2) using spark-submit so i gave in

the main promise of 2) is tha you can provide an application that can run
on multiple hadoop and spark versions. however for that to become true
spark needs to address the issue of user-classpath-first being broken. for
example if i want to use a recent version of avro i am out of luck, even if
i bundle it with my app, because an old version could be in classpath in
spark (it is for example on cdh) and i cannot override the classpath
currently.


On Sun, Jul 27, 2014 at 9:32 PM, Mayur Rustagi <ma...@gmail.com>
wrote:

> Based on some discussions with my application users, I have been trying to
> come up with a standard way to deploy applications built on Spark
>
> 1. Bundle the version of spark with your application and ask users store
> it in hdfs before referring it in yarn to boot your application
> 2. Provide ways to manage dependency in your app across various versions
> of spark bundled in with Hadoop distributions
>
> 1 provides greater control and reliability as I am only working against
> yarn versions and dependencies, I assume 2 gives me some benefits of
> distribution versions of spark (easier management, common sysops tools ?? )
> .
> I was wondering if anyone has thoughts around both and any reasons to
> prefer one over the other.
>
> Sent from my iPad