You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Mendelson, Assaf" <As...@rsa.com> on 2016/10/06 15:46:50 UTC
spark standalone with multiple workers gives a warning
Hi,
I have a spark standalone cluster. On it, I am using 3 workers per node.
So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh
The problem is, that when I run spark-shell I get the following warning:
WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '3').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the spark config.
So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only way I see to start the standalone cluster and the only way I see to define it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES and spark.executor.instances are all related to submitting the job.
Any ideas?
Thanks
Assaf
RE: spark standalone with multiple workers gives a warning
Posted by "Mendelson, Assaf" <As...@rsa.com>.
I am using the script in sbin to set it up (spark/sbin/start-all.sh). It works fine. The problem is how to configure more than one worker per node (the default is one worker only). The documentation for 1.6.1 suggested SPARK_WORKER_INSTANCES as the way to do it but the latest documentation has no option for it.
The warning does not appear when I start the cluster but rather when I submit an application.
I see no way to define SPARK_WORKER_INSTANCES when starting the cluster but not define it when submitting an application.
Assaf.
From: Ofer Eliassaf [mailto:ofer.eliassaf@gmail.com]
Sent: Thursday, October 06, 2016 9:07 PM
To: Mendelson, Assaf
Cc: user@spark.apache.org
Subject: Re: spark standalone with multiple workers gives a warning
The slaves should connect to the master using the scripts in sbin...
You can read about it here:
http://spark.apache.org/docs/latest/spark-standalone.html
On Thu, Oct 6, 2016 at 6:46 PM, Mendelson, Assaf <As...@rsa.com>> wrote:
Hi,
I have a spark standalone cluster. On it, I am using 3 workers per node.
So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh
The problem is, that when I run spark-shell I get the following warning:
WARN SparkConf:
SPARK_WORKER_INSTANCES was detected (set to '3').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --num-executors to specify the number of executors
- Or set SPARK_EXECUTOR_INSTANCES
- spark.executor.instances to configure the number of instances in the spark config.
So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only way I see to start the standalone cluster and the only way I see to define it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES and spark.executor.instances are all related to submitting the job.
Any ideas?
Thanks
Assaf
--
Regards,
Ofer Eliassaf
Re: spark standalone with multiple workers gives a warning
Posted by Ofer Eliassaf <of...@gmail.com>.
The slaves should connect to the master using the scripts in sbin...
You can read about it here:
http://spark.apache.org/docs/latest/spark-standalone.html
On Thu, Oct 6, 2016 at 6:46 PM, Mendelson, Assaf <As...@rsa.com>
wrote:
> Hi,
>
> I have a spark standalone cluster. On it, I am using 3 workers per node.
>
> So I added SPARK_WORKER_INSTANCES set to 3 in spark-env.sh
>
> The problem is, that when I run spark-shell I get the following warning:
>
> WARN SparkConf:
>
> SPARK_WORKER_INSTANCES was detected (set to '3').
>
> This is deprecated in Spark 1.0+.
>
>
>
> Please instead use:
>
> - ./spark-submit with --num-executors to specify the number of executors
>
> - Or set SPARK_EXECUTOR_INSTANCES
>
> - spark.executor.instances to configure the number of instances in the
> spark config.
>
>
>
> So how would I start a cluster of 3? SPARK_WORKER_INSTANCES is the only
> way I see to start the standalone cluster and the only way I see to define
> it is in spark-env.sh. The spark submit option, SPARK_EXECUTOR_INSTANCES
> and spark.executor.instances are all related to submitting the job.
>
>
>
> Any ideas?
>
> Thanks
>
> Assaf
>
--
Regards,
Ofer Eliassaf