You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by robin_up <ro...@gmail.com> on 2014/02/11 03:10:02 UTC
Connecting App to cluster VS Launching app within cluster
Hi
Could someone kindly explain what the pros/cons are between "Connecting an
Application to a cluster" and "Launching Applications inside a cluster" in a
standalone cluster?
I'm referring to the sections in the below doc. I.e. in which situation one
maybe more preferred than the other, or does it matter?
http://spark.incubator.apache.org/docs/latest/spark-standalone.html
thanks
Robin
-----
-- Robin Li
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Connecting App to cluster VS Launching app within cluster
Posted by robin_up <ro...@gmail.com>.
Thanks TD. This is very useful information.
On Thu, Feb 13, 2014 at 12:30 PM, Tathagata Das [via Apache Spark User
List] <ml...@n3.nabble.com> wrote:
> Launching your application in a cluster may be useful in a number of
> scenarios.
>
> 1) In a number of settings in companies, user who want to run jobs do not
> have ssh access to any of the cluster nodes. So they have to run the Spark
> driver program on their "local machine" and connect to the Spark standalone
> cluster's master for launching jobs. That can be pretty slow and can go
> down if the connection between "local machine" and cluster is severed.
> Instead, launching applciation inside the cluster allows you to launch the
> job and forget it.
>
> 2) Another additional functionality that launching in the cluster provides
> is automatic restart of the Spark driver if it fails. This is useful for
> long running Spark jobs (e.g., Spark Streaming job) where you want to
> minimize downtime.
>
> TD
>
>
> On Mon, Feb 10, 2014 at 6:10 PM, robin_up <[hidden email]<http://user/SendEmail.jtp?type=node&node=1481&i=0>
> > wrote:
>
>> Hi
>>
>> Could someone kindly explain what the pros/cons are between "Connecting an
>> Application to a cluster" and "Launching Applications inside a cluster"
>> in a
>> standalone cluster?
>>
>> I'm referring to the sections in the below doc. I.e. in which situation
>> one
>> maybe more preferred than the other, or does it matter?
>>
>> http://spark.incubator.apache.org/docs/latest/spark-standalone.html
>>
>> thanks
>> Robin
>>
>>
>>
>>
>> -----
>> -- Robin Li
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375p1481.html
> To unsubscribe from Connecting App to cluster VS Launching app within
> cluster, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1375&code=cm9iaW4udXBAZ21haWwuY29tfDEzNzV8LTUwNTAwNDExNg==>
> .
> NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Robin
-----
-- Robin Li
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375p1482.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Connecting App to cluster VS Launching app within cluster
Posted by Tathagata Das <ta...@gmail.com>.
Launching your application in a cluster may be useful in a number of
scenarios.
1) In a number of settings in companies, user who want to run jobs do not
have ssh access to any of the cluster nodes. So they have to run the Spark
driver program on their "local machine" and connect to the Spark standalone
cluster's master for launching jobs. That can be pretty slow and can go
down if the connection between "local machine" and cluster is severed.
Instead, launching applciation inside the cluster allows you to launch the
job and forget it.
2) Another additional functionality that launching in the cluster provides
is automatic restart of the Spark driver if it fails. This is useful for
long running Spark jobs (e.g., Spark Streaming job) where you want to
minimize downtime.
TD
On Mon, Feb 10, 2014 at 6:10 PM, robin_up <ro...@gmail.com> wrote:
> Hi
>
> Could someone kindly explain what the pros/cons are between "Connecting an
> Application to a cluster" and "Launching Applications inside a cluster" in
> a
> standalone cluster?
>
> I'm referring to the sections in the below doc. I.e. in which situation one
> maybe more preferred than the other, or does it matter?
>
> http://spark.incubator.apache.org/docs/latest/spark-standalone.html
>
> thanks
> Robin
>
>
>
>
> -----
> -- Robin Li
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Connecting-App-to-cluster-VS-Launching-app-within-cluster-tp1375.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>