You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Manuel Sopena Ballesteros <ma...@garvan.org.au> on 2019/10/10 06:42:26 UTC
can't start spark thrift after Configuring YARN container executor
Dear Hadoop community,
I am trying to configure yarn container executor following this document https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html
I follow all the steps but after restart YARN I can't start spark thrift server.
This is the error I can see in yarn
Application application_1570681803028_0018 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1570681803028_0018_000001 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-10-10 16:49:35.322]Application application_1570681803028_0018 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is spark main : requested yarn user is spark Can't create directory /d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Did not create any app directories For more detailed output, check the application tracking page: http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018 Then click on links to logs of each attempt. . Failing the application.
This is the content of container-executor.cfg
[luffy@gl-hdp-ctrl01-mlx ~]$ cat /etc/hadoop/3.1.0.0-78/0/container-executor.cfg
#/*
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements. See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership. The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License. You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local
yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=hdfs,yarn,mapred,bin
# min.user.id=1000
min.user.id=80
[docker]
module.enabled=false
docker.binary=/usr/bin/docker
docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
docker.allowed.devices=
docker.allowed.networks=host,none,bridge
docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,
docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log,
docker.privileged-containers.enabled=false
docker.trusted.registries=
docker.allowed.volume-drivers=
[gpu]
module.enabled=false
[cgroups]
root=
yarn-hierarchy=
I was hopping if someone could help me troubleshooting about what YARN is trying to do and how to fix this configuration issue?
Thank you very much
Manuel
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
RE: can't start spark thrift after Configuring YARN container
executor
Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Thank you very much Prabhu,
Deleting the /d0 folder fixed the issue
Manuel
From: Prabhu Josephraj [mailto:pjoseph@cloudera.com]
Sent: Thursday, October 10, 2019 6:17 PM
To: Manuel Sopena Ballesteros
Cc: user@hadoop.apache.org
Subject: Re: can't start spark thrift after Configuring YARN container executor
As per the error, spark user does not have permission to create directory under NodeManager Local Directory or the existing spark user directory is with stale uid or gid.
Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018
1. Check if spark user is able to create directory under NM local dir.
2. Remove /d1/hadoop/yarn/local/usercache/spark from all NMs and rerun the job.
On Thu, Oct 10, 2019 at 12:13 PM Manuel Sopena Ballesteros <ma...@garvan.org.au>> wrote:
Dear Hadoop community,
I am trying to configure yarn container executor following this document https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html
I follow all the steps but after restart YARN I can’t start spark thrift server.
This is the error I can see in yarn
Application application_1570681803028_0018 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1570681803028_0018_000001 exited with exitCode: -1000 Failing this attempt.Diagnostics: [2019-10-10 16:49:35.322]Application application_1570681803028_0018 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is spark main : requested yarn user is spark Can't create directory /d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Can't create directory /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018 - Permission denied Did not create any app directories For more detailed output, check the application tracking page: http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018 Then click on links to logs of each attempt. . Failing the application.
This is the content of container-executor.cfg
[luffy@gl-hdp-ctrl01-mlx ~]$ cat /etc/hadoop/3.1.0.0-78/0/container-executor.cfg
#/*
# * Licensed to the Apache Software Foundation (ASF) under one
# * or more contributor license agreements. See the NOTICE file
# * distributed with this work for additional information
# * regarding copyright ownership. The ASF licenses this file
# * to you under the Apache License, Version 2.0 (the
# * "License"); you may not use this file except in compliance
# * with the License. You may obtain a copy of the License at
# *
# * http://www.apache.org/licenses/LICENSE-2.0
# *
# * Unless required by applicable law or agreed to in writing, software
# * distributed under the License is distributed on an "AS IS" BASIS,
# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# * See the License for the specific language governing permissions and
# * limitations under the License.
# */
yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local
yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log
yarn.nodemanager.linux-container-executor.group=hadoop
banned.users=hdfs,yarn,mapred,bin
# min.user.id<http://min.user.id>=1000
min.user.id<http://min.user.id>=80
[docker]
module.enabled=false
docker.binary=/usr/bin/docker
docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
docker.allowed.devices=
docker.allowed.networks=host,none,bridge
docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,
docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log,
docker.privileged-containers.enabled=false
docker.trusted.registries=
docker.allowed.volume-drivers=
[gpu]
module.enabled=false
[cgroups]
root=
yarn-hierarchy=
I was hopping if someone could help me troubleshooting about what YARN is trying to do and how to fix this configuration issue?
Thank you very much
Manuel
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Re: can't start spark thrift after Configuring YARN container executor
Posted by Prabhu Josephraj <pj...@cloudera.com.INVALID>.
As per the error, spark user does not have permission to create directory
under NodeManager Local Directory or the existing spark user directory is
with stale uid or gid.
*Permission denied Can't create directory
/d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018*
1. Check if spark user is able to create directory under NM local dir.
2. Remove /d1/hadoop/yarn/local/usercache/spark from all NMs and rerun the
job.
On Thu, Oct 10, 2019 at 12:13 PM Manuel Sopena Ballesteros <
manuel.sb@garvan.org.au> wrote:
> Dear Hadoop community,
>
>
>
> I am trying to configure yarn container executor following this document
> https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.install.doc/doc/inst_adv_yarn_config.html
>
>
>
> I follow all the steps but after restart YARN I can’t start spark thrift
> server.
>
>
>
> This is the error I can see in yarn
>
>
>
> Application application_1570681803028_0018 failed 1 times (global limit
> =2; local limit is =1) due to AM Container for
> appattempt_1570681803028_0018_000001 exited with exitCode: -1000 Failing
> this attempt.Diagnostics: [2019-10-10 16:49:35.322]Application
> application_1570681803028_0018 initialization failed (exitCode=255) with
> output: main : command provided 0 main : run as user is spark main :
> requested yarn user is spark Can't create directory
> /d0/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018
> - Permission denied Can't create directory
> /d1/hadoop/yarn/local/usercache/spark/appcache/application_1570681803028_0018
> - Permission denied Did not create any app directories For more detailed
> output, check the application tracking page:
> http://gl-hdp-ctrl03-mlx.mlx:8088/cluster/app/application_1570681803028_0018
> Then click on links to logs of each attempt. . Failing the application.
>
>
>
>
>
> This is the content of container-executor.cfg
>
>
>
> [luffy@gl-hdp-ctrl01-mlx ~]$ cat
> /etc/hadoop/3.1.0.0-78/0/container-executor.cfg
>
>
>
>
>
> #/*
>
> # * Licensed to the Apache Software Foundation (ASF) under one
>
> # * or more contributor license agreements. See the NOTICE file
>
> # * distributed with this work for additional information
>
> # * regarding copyright ownership. The ASF licenses this file
>
> # * to you under the Apache License, Version 2.0 (the
>
> # * "License"); you may not use this file except in compliance
>
> # * with the License. You may obtain a copy of the License at
>
> # *
>
> # * http://www.apache.org/licenses/LICENSE-2.0
>
> # *
>
> # * Unless required by applicable law or agreed to in writing, software
>
> # * distributed under the License is distributed on an "AS IS" BASIS,
>
> # * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
>
> # * See the License for the specific language governing permissions and
>
> # * limitations under the License.
>
> # */
>
> yarn.nodemanager.local-dirs=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local
>
> yarn.nodemanager.log-dirs=/d0/hadoop/yarn/log,/d1/hadoop/yarn/log
>
> yarn.nodemanager.linux-container-executor.group=hadoop
>
> banned.users=hdfs,yarn,mapred,bin
>
> # min.user.id=1000
>
> min.user.id=80
>
>
>
> [docker]
>
> module.enabled=false
>
> docker.binary=/usr/bin/docker
>
>
> docker.allowed.capabilities=CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
>
> docker.allowed.devices=
>
> docker.allowed.networks=host,none,bridge
>
> docker.allowed.ro-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,
>
>
> docker.allowed.rw-mounts=/d0/hadoop/yarn/local,/d1/hadoop/yarn/local,/d0/hadoop/yarn/log,/d1/hadoop/yarn/log,
>
> docker.privileged-containers.enabled=false
>
> docker.trusted.registries=
>
> docker.allowed.volume-drivers=
>
>
>
> [gpu]
>
> module.enabled=false
>
>
>
> [cgroups]
>
> root=
>
> yarn-hierarchy=
>
>
>
> I was hopping if someone could help me troubleshooting about what YARN is
> trying to do and how to fix this configuration issue?
>
>
>
> Thank you very much
>
>
>
> Manuel
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>