[jira] [Commented] (YARN-8667) Container Relaunch fails with "find: File system loop detected;" for tar ball artifacts

["Rohith Sharma K S (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580751#comment-16580751 ] 

Rohith Sharma K S commented on YARN-8667:
-----------------------------------------

Container Relaunch shares same working directory. As a result, launch container script create a symlink which is already created leading this issue. In order to debug the issue, follow the below steps
 # launch a sleeper service with below spec. Note that I am providing TAR BALL artifacts
{code}
curl --negotiate -u: -H "Content-Type: application/json" -X POST http://localhost:8088/app/v1/services?user.name=yarn-ats -d '{
  "name": "sleeper",
  "version": "1.0.0",
  "queue": "default",
  "artifact": {
    "id": "/mapreduce/mapreduce.tar.gz",
    "type": "TARBALL"
  },
  "components" :
  [
    {
      "name": "sleeper1",
      "number_of_containers": 1,
      "launch_command": "sleep infinity",
      "resource": {
        "cpus": 1,
        "memory": "2048"
      }
    }
  ]
}'
{code}
# After sleeper service is launched, goto working directory of container. There you see below files
{noformat}
[root@ctr-e138-1518143905142-431547-01-000004 container_e04_1534244457405_0004_01_000002]# ll
total 24
-rw-r--r-- 1 yarn hadoop    7 Aug 15 05:47 container_tokens
-rwx------ 1 yarn hadoop  656 Aug 15 05:47 default_container_executor_session.sh
-rwx------ 1 yarn hadoop  711 Aug 15 05:47 default_container_executor.sh
-rwx------ 1 yarn hadoop 3817 Aug 15 05:47 launch_container.sh
lrwxrwxrwx 1 yarn hadoop  107 Aug 15 05:47 lib -> /hadoop/yarn/local/usercache/yarn-ats/appcache/application_1534244457405_0004/filecache/10/mapreduce.tar.gz
drwx--x--- 2 yarn hadoop 4096 Aug 15 05:47 tmp
{noformat}
# You can try to execute launch_container.sh again manually which fails with below error.
{code:}
find: File system loop detected; ‘./lib/mapreduce.tar.gz’ is part of the same file system loop as ‘./lib’.{code}

In container relaunch also the same working directory is shared which executes launch_container.sh again. This is causing an error which terminates launch_container.sh with error code 1

> Container Relaunch fails with "find: File system loop detected;" for tar ball artifacts
> ---------------------------------------------------------------------------------------
>
>                 Key: YARN-8667
>                 URL: https://issues.apache.org/jira/browse/YARN-8667
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Rohith Sharma K S
>            Priority: Major
>
> Service is launched with TAR BALL artifacts. If a container is exited due to any reasons, container relaunch policy try to relaunch the container on same node with same container work space. As a result, container relaunch is keep on failing. 
> If container relaunch max-retry policy is disabled, then  container never launched in any other nodes also rather it keep on retrying on same node manager which never succeeds.
> {code}
> Relaunching Container container_e05_1533635581781_0001_01_000002. Remaining retry attempts(after relaunch) : -4816.
> {code}
> There are two issues
> # Container relaunch is keep on failing
> # Log message is misleading



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org