apache-spark-how to run distributed spark jobs in Apache-spark-standalone-cluster

[Dinakar Chennubotla <ch...@gmail.com>]

Hi All,i am dinakar and I am admin,i have question,
Question: " is it possible to run distributed spark jobs in Apache spark
standalone cluster".if yes, could someone, help with the docs or webpages.
so that i can create and test it.Thanks in advance,
Dinakar

Re: apache-spark-how to run distributed spark jobs in Apache-spark-standalone-cluster

[Dinakar Chennubotla <ch...@gmail.com>]

Hi Sean Owen,

Thank you very much for your quick response.
Could you please help us, with the below?


coming to the point, yes I did check the link you sent. but I did not get
whole clarity, what I am searching for.

Our Prod problem statement is:
=========================
1. Can we launch distributed way of spark jobs in the Spark standalone
cluster, if yes, how.

our Prod  details:
==============
1. we have a spark cluster, standalone deployment done, all are running
inside docker containers, no orchestrator or monitoring frame deployed.
2. where 1 node has a spark master, the rest spark workers are deployed
inside a docker.


with the above details mentioned, we implement the below rules and launch
spark jobs in a distributed way in the "spark standalone cluster ".
Rule 1:
 the first rule is that,  if start launching the spark jobs, jobs should be
launched in the ideal spark-worker node.

Rule 2:
IF no ideal spark-worker node is available, then
resources-availability should be checked in the current cluster and then
launch the jobs in the resources-available node.


Thanks and regards,
Dinakar

On Mon, Nov 8, 2021 at 6:12 PM Sean Owen <sr...@gmail.com> wrote:

> Yes, did you check the docs?
> https://spark.apache.org/docs/latest/spark-standalone.html
>
> On Mon, Nov 8, 2021 at 6:40 AM Dinakar Chennubotla <
> chennu.bigdata@gmail.com> wrote:
>
>> Hi All,i am dinakar and I am admin,i have question,
>> Question: " is it possible to run distributed spark jobs in Apache spark
>> standalone cluster".if yes, could someone, help with the docs or
>> webpages. so that i can create and test it.Thanks in advance,
>> Dinakar
>>
>

Re: apache-spark-how to run distributed spark jobs in Apache-spark-standalone-cluster

[Sean Owen <sr...@gmail.com>]

Yes, did you check the docs?
https://spark.apache.org/docs/latest/spark-standalone.html

On Mon, Nov 8, 2021 at 6:40 AM Dinakar Chennubotla <ch...@gmail.com>
wrote:

> Hi All,i am dinakar and I am admin,i have question,
> Question: " is it possible to run distributed spark jobs in Apache spark
> standalone cluster".if yes, could someone, help with the docs or
> webpages. so that i can create and test it.Thanks in advance,
> Dinakar
>