You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by davvy benny <da...@gmail.com> on 2021/09/27 21:37:30 UTC
21/09/27 23:34:03 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Hi
I am trying to run spark programmatically from eclipse with these configurations for hadoop cluster locally
SparkConf sparkConf = new SparkConf().setAppName("simpleTest2").setMaster("yarn")
.set("spark.executor.memory", "1g")
.set("deploy.mode", "cluster")
.set("spark.yarn.stagingDir", "hdfs://localhost:9000/user/hadoop/")
.set("spark.shuffle.service.enabled", "false")
.set("spark.dynamicAllocation.enabled", "false")
.set("spark.cores.max", "1")
.set("spark.executor.instances","2")
.set("spark.executor.memory","500m") //
.set("spark.executor.cores","1")//
.set("spark.yarn.nodemanager.resource.cpu-vcores","4")
.set("spark.yarn.submit.file.replication", "1")
.set("spark.yarn.jars", "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
When I check on the http://localhost:8088/cluster/apps/RUNNING I can see that my job is submitted but y terminal loops saying
21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
I ve noticed that this occurs after the application of a map on my Dataset.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Posted by davvy benny <da...@gmail.com>.
Thank you For the answer
I have set now these properties as you suggest
SparkConf sparkConf = new SparkConf().setAppName("simpleTest2").setMaster("yarn")
.set("spark.executor.memory", "1g")
.set("deploy.mode", "cluster")
.set("spark.yarn.stagingDir", "hdfs://localhost:9000/user/hadoop/")
.set("spark.hadoop.fs.defaultFS","hdfs://localhost:9000")
.set("spark.dynamicAllocation.enabled", "false")
.set("spark.worker.cores","1")
.set("maximizeResourceAllocation", "true")
.set("spark.executor.memory","1g") //
.set("spark.executor.cores","1")//
.set("spark.worker.instances","1")
.set("spark.driver.instances","1")
.set("spark.driver.cores","1")
.set("spark.driver.memory","1g")
I have Checked at localhost:8080 I have 1 worker active (I have activated with spark/sbin/start-worker.sh spark://localhost:7077). I guess whether I need this worker active. Because when i launch my job its status does not change
On 2021/09/28 12:29:24, Stelios Philippou <st...@gmail.com> wrote:
> You need to check your Spark Cluster.
> I can assume that this is local
>
> http://localhost:8080
>
> As the example below you will need to have 2 worker instances and 1 driver
> instance so most probably 3 workers.
> They need to have at least one Core and 1g ram to them. But you must not
> use all the available ram. Aim for using 70% of the available ram at the
> beginning.
>
> You are also adding the
> > > > > .set("spark.executor.memory", "1g")
> twice. Perhaps you need to driver instance. ?
>
> An example would bem but you can translate them to SparkConf
>
> --conf spark.executor.cores=1 \
>
> --conf spark.executor.memory=1g \
>
>
> --conf spark.driver.instances=1 \
>
> --conf spark.driver.cores=1 \
>
> --conf spark.driver.memory=1g \
>
>
> Also setting the
> > > > > .set("spark.executor.instances","2")
> might not work and the executor would pick up all available instances.
>
> [image: image.png]
>
> On Tue, 28 Sept 2021 at 15:09, davvy benny <da...@gmail.com> wrote:
>
> > How can I check it?
> >
> > On 2021/09/28 03:29:45, Stelios Philippou <st...@gmail.com> wrote:
> > > It might be possible that you do not have the resources on the cluster.
> > So
> > > your job will remain to wait for them as they cannot be provided.
> > >
> > > On Tue, 28 Sep 2021, 04:26 davvy benny, <da...@gmail.com> wrote:
> > >
> > > > How can I solve the problem?
> > > >
> > > > On 2021/09/27 23:05:41, Thejdeep G <te...@gmail.com> wrote:
> > > > > Hi,
> > > > >
> > > > > That would usually mean that the application has not been allocated
> > the
> > > > executor resources from the resource manager yet.
> > > > >
> > > > > On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> > > > > > Hi
> > > > > > I am trying to run spark programmatically from eclipse with these
> > > > configurations for hadoop cluster locally
> > > > > > SparkConf sparkConf = new
> > > > SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > > > > > .set("spark.executor.memory", "1g")
> > > > > > .set("deploy.mode", "cluster")
> > > > > > .set("spark.yarn.stagingDir",
> > > > "hdfs://localhost:9000/user/hadoop/")
> > > > > > .set("spark.shuffle.service.enabled", "false")
> > > > > > .set("spark.dynamicAllocation.enabled",
> > "false")
> > > > > > .set("spark.cores.max", "1")
> > > > > > .set("spark.executor.instances","2")
> > > > > > .set("spark.executor.memory","500m") //
> > > > > > .set("spark.executor.cores","1")//
> > > > > >
> > > > .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > > > > >
> > .set("spark.yarn.submit.file.replication",
> > > > "1")
> > > > > > .set("spark.yarn.jars",
> > > > "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> > > > > >
> > > > > > When I check on the http://localhost:8088/cluster/apps/RUNNING I
> > can
> > > > see that my job is submitted but y terminal loops saying
> > > > > > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted
> > any
> > > > resources; check your cluster UI to ensure that workers are registered
> > and
> > > > have sufficient resources
> > > > > >
> > > > > > I ve noticed that this occurs after the application of a map on my
> > > > Dataset.
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient resources
Posted by Stelios Philippou <st...@gmail.com>.
You need to check your Spark Cluster.
I can assume that this is local
http://localhost:8080
As the example below you will need to have 2 worker instances and 1 driver
instance so most probably 3 workers.
They need to have at least one Core and 1g ram to them. But you must not
use all the available ram. Aim for using 70% of the available ram at the
beginning.
You are also adding the
> > > > .set("spark.executor.memory", "1g")
twice. Perhaps you need to driver instance. ?
An example would bem but you can translate them to SparkConf
--conf spark.executor.cores=1 \
--conf spark.executor.memory=1g \
--conf spark.driver.instances=1 \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=1g \
Also setting the
> > > > .set("spark.executor.instances","2")
might not work and the executor would pick up all available instances.
[image: image.png]
On Tue, 28 Sept 2021 at 15:09, davvy benny <da...@gmail.com> wrote:
> How can I check it?
>
> On 2021/09/28 03:29:45, Stelios Philippou <st...@gmail.com> wrote:
> > It might be possible that you do not have the resources on the cluster.
> So
> > your job will remain to wait for them as they cannot be provided.
> >
> > On Tue, 28 Sep 2021, 04:26 davvy benny, <da...@gmail.com> wrote:
> >
> > > How can I solve the problem?
> > >
> > > On 2021/09/27 23:05:41, Thejdeep G <te...@gmail.com> wrote:
> > > > Hi,
> > > >
> > > > That would usually mean that the application has not been allocated
> the
> > > executor resources from the resource manager yet.
> > > >
> > > > On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> > > > > Hi
> > > > > I am trying to run spark programmatically from eclipse with these
> > > configurations for hadoop cluster locally
> > > > > SparkConf sparkConf = new
> > > SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > > > > .set("spark.executor.memory", "1g")
> > > > > .set("deploy.mode", "cluster")
> > > > > .set("spark.yarn.stagingDir",
> > > "hdfs://localhost:9000/user/hadoop/")
> > > > > .set("spark.shuffle.service.enabled", "false")
> > > > > .set("spark.dynamicAllocation.enabled",
> "false")
> > > > > .set("spark.cores.max", "1")
> > > > > .set("spark.executor.instances","2")
> > > > > .set("spark.executor.memory","500m") //
> > > > > .set("spark.executor.cores","1")//
> > > > >
> > > .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > > > >
> .set("spark.yarn.submit.file.replication",
> > > "1")
> > > > > .set("spark.yarn.jars",
> > > "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> > > > >
> > > > > When I check on the http://localhost:8088/cluster/apps/RUNNING I
> can
> > > see that my job is submitted but y terminal loops saying
> > > > > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted
> any
> > > resources; check your cluster UI to ensure that workers are registered
> and
> > > have sufficient resources
> > > > >
> > > > > I ve noticed that this occurs after the application of a map on my
> > > Dataset.
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Posted by davvy benny <da...@gmail.com>.
How can I check it?
On 2021/09/28 03:29:45, Stelios Philippou <st...@gmail.com> wrote:
> It might be possible that you do not have the resources on the cluster. So
> your job will remain to wait for them as they cannot be provided.
>
> On Tue, 28 Sep 2021, 04:26 davvy benny, <da...@gmail.com> wrote:
>
> > How can I solve the problem?
> >
> > On 2021/09/27 23:05:41, Thejdeep G <te...@gmail.com> wrote:
> > > Hi,
> > >
> > > That would usually mean that the application has not been allocated the
> > executor resources from the resource manager yet.
> > >
> > > On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> > > > Hi
> > > > I am trying to run spark programmatically from eclipse with these
> > configurations for hadoop cluster locally
> > > > SparkConf sparkConf = new
> > SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > > > .set("spark.executor.memory", "1g")
> > > > .set("deploy.mode", "cluster")
> > > > .set("spark.yarn.stagingDir",
> > "hdfs://localhost:9000/user/hadoop/")
> > > > .set("spark.shuffle.service.enabled", "false")
> > > > .set("spark.dynamicAllocation.enabled", "false")
> > > > .set("spark.cores.max", "1")
> > > > .set("spark.executor.instances","2")
> > > > .set("spark.executor.memory","500m") //
> > > > .set("spark.executor.cores","1")//
> > > >
> > .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > > > .set("spark.yarn.submit.file.replication",
> > "1")
> > > > .set("spark.yarn.jars",
> > "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> > > >
> > > > When I check on the http://localhost:8088/cluster/apps/RUNNING I can
> > see that my job is submitted but y terminal loops saying
> > > > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any
> > resources; check your cluster UI to ensure that workers are registered and
> > have sufficient resources
> > > >
> > > > I ve noticed that this occurs after the application of a map on my
> > Dataset.
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient resources
Posted by Stelios Philippou <st...@gmail.com>.
It might be possible that you do not have the resources on the cluster. So
your job will remain to wait for them as they cannot be provided.
On Tue, 28 Sep 2021, 04:26 davvy benny, <da...@gmail.com> wrote:
> How can I solve the problem?
>
> On 2021/09/27 23:05:41, Thejdeep G <te...@gmail.com> wrote:
> > Hi,
> >
> > That would usually mean that the application has not been allocated the
> executor resources from the resource manager yet.
> >
> > On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> > > Hi
> > > I am trying to run spark programmatically from eclipse with these
> configurations for hadoop cluster locally
> > > SparkConf sparkConf = new
> SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > > .set("spark.executor.memory", "1g")
> > > .set("deploy.mode", "cluster")
> > > .set("spark.yarn.stagingDir",
> "hdfs://localhost:9000/user/hadoop/")
> > > .set("spark.shuffle.service.enabled", "false")
> > > .set("spark.dynamicAllocation.enabled", "false")
> > > .set("spark.cores.max", "1")
> > > .set("spark.executor.instances","2")
> > > .set("spark.executor.memory","500m") //
> > > .set("spark.executor.cores","1")//
> > >
> .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > > .set("spark.yarn.submit.file.replication",
> "1")
> > > .set("spark.yarn.jars",
> "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> > >
> > > When I check on the http://localhost:8088/cluster/apps/RUNNING I can
> see that my job is submitted but y terminal loops saying
> > > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient resources
> > >
> > > I ve noticed that this occurs after the application of a map on my
> Dataset.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Posted by davvy benny <da...@gmail.com>.
How can I solve the problem?
On 2021/09/27 23:05:41, Thejdeep G <te...@gmail.com> wrote:
> Hi,
>
> That would usually mean that the application has not been allocated the executor resources from the resource manager yet.
>
> On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> > Hi
> > I am trying to run spark programmatically from eclipse with these configurations for hadoop cluster locally
> > SparkConf sparkConf = new SparkConf().setAppName("simpleTest2").setMaster("yarn")
> > .set("spark.executor.memory", "1g")
> > .set("deploy.mode", "cluster")
> > .set("spark.yarn.stagingDir", "hdfs://localhost:9000/user/hadoop/")
> > .set("spark.shuffle.service.enabled", "false")
> > .set("spark.dynamicAllocation.enabled", "false")
> > .set("spark.cores.max", "1")
> > .set("spark.executor.instances","2")
> > .set("spark.executor.memory","500m") //
> > .set("spark.executor.cores","1")//
> > .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> > .set("spark.yarn.submit.file.replication", "1")
> > .set("spark.yarn.jars", "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
> >
> > When I check on the http://localhost:8088/cluster/apps/RUNNING I can see that my job is submitted but y terminal loops saying
> > 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
> >
> > I ve noticed that this occurs after the application of a map on my Dataset.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: 21/09/27 23:34:03 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
Posted by Thejdeep G <te...@gmail.com>.
Hi,
That would usually mean that the application has not been allocated the executor resources from the resource manager yet.
On 2021/09/27 21:37:30, davvy benny <da...@gmail.com> wrote:
> Hi
> I am trying to run spark programmatically from eclipse with these configurations for hadoop cluster locally
> SparkConf sparkConf = new SparkConf().setAppName("simpleTest2").setMaster("yarn")
> .set("spark.executor.memory", "1g")
> .set("deploy.mode", "cluster")
> .set("spark.yarn.stagingDir", "hdfs://localhost:9000/user/hadoop/")
> .set("spark.shuffle.service.enabled", "false")
> .set("spark.dynamicAllocation.enabled", "false")
> .set("spark.cores.max", "1")
> .set("spark.executor.instances","2")
> .set("spark.executor.memory","500m") //
> .set("spark.executor.cores","1")//
> .set("spark.yarn.nodemanager.resource.cpu-vcores","4")
> .set("spark.yarn.submit.file.replication", "1")
> .set("spark.yarn.jars", "hdfs://localhost:9000/user/hadoop/davben/jars/*.jar")
>
> When I check on the http://localhost:8088/cluster/apps/RUNNING I can see that my job is submitted but y terminal loops saying
> 21/09/27 23:36:33 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
>
> I ve noticed that this occurs after the application of a map on my Dataset.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org