You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@myriad.apache.org by sarjeet singh <ss...@gmail.com> on 2016/05/23 03:54:08 UTC
Issues related to RM docker container on Myriad RC2.
I tried following to try out docker from myriad rc2, but couldn't past
after RM is launched and not able to launch NMs.
Here is the formatted output for the RM docker launch:
root@qa101-139:~/myriad/myriad-0.2.0-incubating-rc2/docker# docker run
--net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop
--name='myriad-resourcemanager' -t sarjeet/myriad
2016-05-23 03:38:21,431 INFO [main] myriad.Main
(Main.java:initHealthChecks(140)) - Initializing HealthChecks
2016-05-23 03:38:21,445 INFO [main] myriad.Main
(Main.java:initProfiles(148)) - Initializing Profiles
2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
(ServiceProfileManager.java:add(40)) - Adding profile zero with CPU: 0.0
and Memory: 0.0
2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
(ServiceProfileManager.java:add(40)) - Adding profile small with CPU: 2.0
and Memory: 2048.0
2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
(ServiceProfileManager.java:add(40)) - Adding profile medium with CPU: 4.0
and Memory: 4096.0
2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
(ServiceProfileManager.java:add(40)) - Adding profile large with CPU: 10.0
and Memory: 12288.0
2016-05-23 03:38:21,451 INFO [main] myriad.Main
(Main.java:validateNMInstances(175)) - Validating nmInstances..
2016-05-23 03:38:21,451 INFO [main] myriad.Main
(Main.java:initServiceConfigurations(238)) - Initializing
initServiceConfigurations
2016-05-23 03:38:21,534 INFO [main] myriad.Main
(Main.java:startMesosDriver(119)) - starting mesosDriver..
2016-05-23 03:38:21,534 INFO [main] scheduler.MyriadDriverManager
(MyriadDriverManager.java:startDriver(51)) - Starting driver...
2016-05-23 03:38:21,534 INFO [main] scheduler.MyriadDriver
(MyriadDriver.java:start(49)) - Starting driver
2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@712: Client
environment:zookeeper.version=zookeeper C client 3.4.5
2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@716: Client
environment:host.name=qa101-139
2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@723: Client
environment:os.name=Linux
2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@724: Client
environment:os.arch=3.13.0-57-generic
2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@725: Client
environment:os.version=#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015
I0523 03:38:21.535646 24 sched.cpp:222] Version: 0.28.1
2016-05-23 03:38:21,535 INFO [main] scheduler.MyriadDriver
(MyriadDriver.java:start(51)) - Driver started with status: DRIVER_RUNNING
2016-05-23 03:38:21,536 INFO [main] scheduler.MyriadDriverManager
(MyriadDriverManager.java:startDriver(53)) - Driver started with status:
DRIVER_RUNNING
2016-05-23 03:38:21,536 INFO [main] myriad.Main
(Main.java:startMesosDriver(121)) - started mesosDriver..
2016-05-23 03:38:21,536:7(0x7f4e8d685700):ZOO_INFO@check_events@1703:
initiated connection to server [10.10.101.139:5181]
2016-05-23 03:38:21,536 INFO [main] interceptor.CompositeInterceptor
(CompositeInterceptor.java:register(74)) - Registered
org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
2016-05-23 03:38:21,539 INFO [main] myriad.Main
(Main.java:startNMInstances(226)) - Launching 1 NM(s) with profile medium
2016-05-23 03:38:21,540 INFO [main] scheduler.MyriadOperations
(MyriadOperations.java:flexUpCluster(80)) - Adding 1 NM instances to cluster
2016-05-23 03:38:21,555:7(0x7f4e8d685700):ZOO_INFO@check_events@1750:
session establishment complete on server [10.10.101.139:5181],
sessionId=0x15314ddb816d02b, negotiated timeout=10000
I0523 03:38:21.555871 97 group.cpp:349] Group process (group(1)@
10.10.101.139:57196) connected to ZooKeeper
I0523 03:38:21.555945 97 group.cpp:831] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0523 03:38:21.555979 97 group.cpp:427] Trying to create path '/mesos'
in ZooKeeper
I0523 03:38:21.557073 98 detector.cpp:152] Detected a new leader:
(id='1')
I0523 03:38:21.557235 75 group.cpp:700] Trying to get
'/mesos/json.info_0000000001' in ZooKeeper
I0523 03:38:21.558002 97 detector.cpp:479] A new leading master (UPID=
master@10.10.101.139:5050) is detected
I0523 03:38:21.558116 76 sched.cpp:326] New master detected at
master@10.10.101.139:5050
I0523 03:38:21.558442 76 sched.cpp:336] No credentials provided.
Attempting to register without authentication
I0523 03:38:21.559672 93 sched.cpp:703] Framework registered with
8114e114-db5f-4faa-afb3-ba1ae29e6368-0023
2016-05-23 03:38:21,630 INFO [pool-2-thread-1]
handlers.RegisteredEventHandler (RegisteredEventHandler.java:onEvent(41)) -
Received event: org.apache.myriad.scheduler.event.RegisteredEvent@27492832
with frameworkId: value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-0023"
2016-05-23 03:38:21,691 INFO [main] state.SchedulerState
(SchedulerState.java:addNodes(77)) - Marked taskId
nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 pending, size of pending
queue for nm is: 0
2016-05-23 03:38:21,702 INFO [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:onEvent(100)) - Received offers 1
2016-05-23 03:38:21,702 INFO [main] scheduler.MyriadOperations
(MyriadOperations.java:flexUpAService(139)) - Adding 1 jobhistory instances
to cluster
2016-05-23 03:38:21,708 INFO [main] state.SchedulerState
(SchedulerState.java:addNodes(77)) - Marked taskId
jobhistory.jobhistory.78b0a6a2-a47d-412f-869d-6fd9872ef85a pending, size of
pending queue for jobhistory is: 0
2016-05-23 03:38:21,713 INFO [main]
interceptor.MyriadInitializationInterceptor
(MyriadInitializationInterceptor.java:init(54)) - Initialized myriad.
2016-05-23 03:38:21,714 WARN [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:matches(198)) - Ignoring unknown resource
type: dfsio_spindles
2016-05-23 03:38:21,724 INFO [pool-2-thread-3]
scheduler.TaskFactory$NMTaskFactoryImpl
(TaskFactory.java:getCommandInfo(138)) - Getting Hadoop distribution from:
http://172.17.0.2:8192/api/config.tgz
2016-05-23 03:38:21,749 INFO [main] ipc.CallQueueManager
(CallQueueManager.java:<init>(53)) - Using callQueue class
java.util.concurrent.LinkedBlockingQueue
2016-05-23 03:38:21,758 INFO [Socket Reader #1 for port 8031] ipc.Server
(Server.java:run(606)) - Starting Socket Reader #1 for port 8031
2016-05-23 03:38:21,774 INFO [main] pb.RpcServerFactoryPBImpl
(RpcServerFactoryPBImpl.java:createServer(174)) - Adding protocol
org.apache.hadoop.yarn.server.api.ResourceTrackerPB to the server
2016-05-23 03:38:21,775 INFO [IPC Server Responder] ipc.Server
(Server.java:run(836)) - IPC Server Responder: starting
2016-05-23 03:38:21,775 INFO [IPC Server listener on 8031] ipc.Server
(Server.java:run(676)) - IPC Server listener on 8031: starting
2016-05-23 03:38:21,786 INFO [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:onEvent(139)) - Launching task:
nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 using offer: value:
"8114e114-db5f-4faa-afb3-ba1ae29e6368-O98094"
2016-05-23 03:38:21,802 INFO [main] ipc.CallQueueManager
(CallQueueManager.java:<init>(53)) - Using callQueue class
java.util.concurrent.LinkedBlockingQueue
2016-05-23 03:38:21,803 INFO [pool-2-thread-9]
handlers.ExecutorLostEventHandler
(ExecutorLostEventHandler.java:onEvent(39)) - Executor value:
"myriad_executor8114e114-db5f-4faa-afb3-ba1ae29e6368-00238114e114-db5f-4faa-afb3-ba1ae29e6368-O980948114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
of slave value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
lost with exit status: -1
2016-05-23 03:38:21,809 INFO [Socket Reader #1 for port 8030] ipc.Server
(Server.java:run(606)) - Starting Socket Reader #1 for port 8030
2016-05-23 03:38:21,824 INFO [pool-2-thread-5]
handlers.StatusUpdateEventHandler
(StatusUpdateEventHandler.java:onEvent(60)) - Status Update for task:
nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 | state: TASK_FAILED
2016-05-23 03:38:21,852 INFO [main] pb.RpcServerFactoryPBImpl
(RpcServerFactoryPBImpl.java:createServer(174)) - Adding protocol
org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB to the server
2016-05-23 03:38:21,853 INFO [IPC Server Responder] ipc.Server
(Server.java:run(836)) - IPC Server Responder: starting
2016-05-23 03:38:21,853 INFO [IPC Server listener on 8030] ipc.Server
(Server.java:run(676)) - IPC Server listener on 8030: starting
2016-05-23 03:38:22,504 INFO [IPC Server listener on 8033] ipc.Server
(Server.java:run(676)) - IPC Server listener on 8033: starting
2016-05-23 03:38:22,755 INFO [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:onEvent(100)) - Received offers 1
2016-05-23 03:38:22,755 WARN [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:matches(198)) - Ignoring unknown resource
type: dfsio_spindles
2016-05-23 03:38:22,756 INFO [pool-2-thread-3]
scheduler.TaskFactory$NMTaskFactoryImpl
(TaskFactory.java:getCommandInfo(138)) - Getting Hadoop distribution from:
http://172.17.0.2:8192/api/config.tgz
2016-05-23 03:38:22,757 INFO [pool-2-thread-3]
handlers.ResourceOffersEventHandler
(ResourceOffersEventHandler.java:onEvent(139)) - Launching task:
nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 using offer: value:
"8114e114-db5f-4faa-afb3-ba1ae29e6368-O98095"
2016-05-23 03:38:22,768 INFO [pool-2-thread-9]
handlers.ExecutorLostEventHandler
(ExecutorLostEventHandler.java:onEvent(39)) - Executor value:
"myriad_executor8114e114-db5f-4faa-afb3-ba1ae29e6368-00238114e114-db5f-4faa-afb3-ba1ae29e6368-O980958114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
of slave value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
lost with exit status: -1
2016-05-23 03:38:22,773 INFO [pool-2-thread-5]
handlers.StatusUpdateEventHandler
(StatusUpdateEventHandler.java:onEvent(60)) - Status Update for task:
nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 | state: TASK_FAILED
====================================
To debug above failure, I checked the NM task stdout/stderr but failed to
get any logs (see screenshot attached)
Then, On attaching to the running containers, found following from the
docker container:
root@qa101-139:~/myriad/myriad-0.2.0-incubating-rc2/docker# docker exec -it
8f730dc2de1a /bin/bash
yarn@qa101-139:/$
yarn@qa101-139:/$ ps -ef
UID PID PPID C STIME TTY TIME CMD
yarn 1 0 0 03:38 ? 00:00:00 /bin/sh -c
/usr/local/hadoop/bin/yarn resourcemanager
yarn 7 1 14 03:38 ? 00:00:14 /usr/bin/java
-Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/usr/local/hadoop/logs
-Dyarn.log.dir=/usr/l
yarn 296 0 0 03:39 ? 00:00:00 /bin/bash
yarn 302 0 0 03:39 ? 00:00:00 /bin/bash
yarn 309 302 0 03:39 ? 00:00:00 ps -ef
yarn@qa101-139:/$ ls -l /usr/local/hadoop/
total 52
-rw-r--r-- 1 root root 15429 Apr 10 2015 LICENSE.txt
-rw-r--r-- 1 root root 101 Apr 10 2015 NOTICE.txt
-rw-r--r-- 1 root root 1366 Apr 10 2015 README.txt
drwxr-xr-x 2 root root 4096 May 22 19:22 bin
drwxr-xr-x 6 root root 4096 May 22 20:42 etc
drwxr-xr-x 2 root root 4096 May 22 19:22 include
drwxr-xr-x 4 root root 4096 May 22 19:22 lib
drwxr-xr-x 2 root root 4096 May 22 19:22 libexec
drwxr-xr-x 2 root root 4096 May 22 19:22 sbin
drwxr-xr-x 7 root root 4096 May 22 20:42 share
yarn@qa101-139:/$ ls -l /usr/local/hadoop/etc/hadoop/
total 16
-rw-r--r-- 1 root root 1340 May 22 20:02 mapred-site.xml
-rw-r--r-- 1 root root 3395 May 23 03:38 myriad-config-default.yml
-rw-r--r-- 1 root root 4207 May 23 00:27 yarn-site.xml
yarn@qa101-139:/$ cat
/usr/local/hadoop/etc/hadoop/myriad-config-default.yml
mesosMaster: zk://10.10.101.139:5181/mesos ->> (Running on the host
outside of the container)
#Container information for the node managers
containerInfo:
type: DOCKER
dockerInfo:
image: sarjeet/myriad
volumes:
-
containerPath: /tmp
hostPath: /tmp
checkpoint: false
frameworkFailoverTimeout: 43200000
frameworkName: MyriadAlpha
frameworkRole:
frameworkUser: mapr
# running the resource manager.
frameworkSuperUser: root # To be depricated, currently permissions need
set by a superuser due to Mesos-1790. Must be
# root or have passwordless sudo. Required if
nodeManagerURI set, ignored otherwise.
nativeLibrary: /usr/local/lib/libmesos.so
zkServers: 10.10.101.139:5181
zkTimeout: 20000
restApiPort: 8192
profiles:
zero: # NMs launched with this profile dynamically obtain cpu/mem from
Mesos
cpu: 0
mem: 0
spindles: 0
small:
cpu: 2
mem: 2048
spindles: 1
medium:
cpu: 4
mem: 4096
spindles: 2
large:
cpu: 10
mem: 12288
spindles: 4
nmInstances: # NMs to start with. Requires at least 1 NM with a non-zero
profile.
medium: 1 # <profile_name : instances>
rebalancer: false
haEnabled: true
servedConfigPath: /dist/config.tgz
nodemanager:
jvmMaxMemoryMB: 1024
cpus: 0.2
cgroups: true
executor:
jvmMaxMemoryMB: 256
configUri: http://172.17.0.1:8192/api/config.tgz
path:
file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.1.0.jar
#The following should be used for a remotely distributed URI, hdfs
assumed but other URI types valid.
#nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz
#path:
file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.1.0.jar
yarnEnvironment:
YARN_NODEMANAGER_OPTS: -Dcluster.name.prefix=/cluster1
-Dnodemanager.resource.io-spindles=4.0
YARN_HOME: /opt/mapr/hadoop/hadoop-2.7.0
HADOOP_CONF_DIR: /mnt/mesos/sandbox/config
HADOOP_TMP_DIR: /tmp
HADOOP_LOG_DIR: /mnt/mesos/sandbox
#JAVA_HOME: /usr/lib/jvm/java-default #System dependent, but sometimes
necessary
mesosAuthenticationPrincipal:
mesosAuthenticationSecretFilename:
yarn@qa101-139:/$ netstat -anlp | grep 8088
tcp6 0 0 10.10.101.139:8088 :::* LISTEN
7/java
yarn@qa101-139:/$ netstat -anlp | grep 8192
tcp6 0 0 :::8192 :::* LISTEN
7/java
yarn@qa101-139:/$
====================================
Reference: https://github.com/apache/incubator-myriad/tree/master/docker
I might not have looked deeper enough to see if there was any configuration
issue on launching docker RM, but in case, there is a trivial fix or config
I missed, I can give this another try. Let me know if there was anything I
missed?
- Sarjeet Singh
Re: Issues related to RM docker container on Myriad RC2.
Posted by Sarjeet Singh <sa...@maprtech.com>.
AFAIK, I might have few configuration and setup issue related to docker
issues. I'll still look into it more once I have some more time. But
overall, RC2 looks good and docker issue is not a blocker as such.
Let me know if anyone else is able to try docker stuff successfully and
willing to share the story :).
-Sarjeet
On Sun, May 22, 2016 at 8:54 PM, sarjeet singh <ss...@gmail.com>
wrote:
> I tried following to try out docker from myriad rc2, but couldn't past
> after RM is launched and not able to launch NMs.
>
> Here is the formatted output for the RM docker launch:
>
> root@qa101-139:~/myriad/myriad-0.2.0-incubating-rc2/docker# docker run
> --net=host -v $PWD/dist -v $PWD/config:/usr/local/hadoop/etc/hadoop
> --name='myriad-resourcemanager' -t sarjeet/myriad
>
> 2016-05-23 03:38:21,431 INFO [main] myriad.Main
> (Main.java:initHealthChecks(140)) - Initializing HealthChecks
>
> 2016-05-23 03:38:21,445 INFO [main] myriad.Main
> (Main.java:initProfiles(148)) - Initializing Profiles
>
> 2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
> (ServiceProfileManager.java:add(40)) - Adding profile zero with CPU: 0.0
> and Memory: 0.0
>
> 2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
> (ServiceProfileManager.java:add(40)) - Adding profile small with CPU: 2.0
> and Memory: 2048.0
>
> 2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
> (ServiceProfileManager.java:add(40)) - Adding profile medium with CPU: 4.0
> and Memory: 4096.0
>
> 2016-05-23 03:38:21,450 INFO [main] scheduler.ServiceProfileManager
> (ServiceProfileManager.java:add(40)) - Adding profile large with CPU: 10.0
> and Memory: 12288.0
>
> 2016-05-23 03:38:21,451 INFO [main] myriad.Main
> (Main.java:validateNMInstances(175)) - Validating nmInstances..
>
> 2016-05-23 03:38:21,451 INFO [main] myriad.Main
> (Main.java:initServiceConfigurations(238)) - Initializing
> initServiceConfigurations
>
> 2016-05-23 03:38:21,534 INFO [main] myriad.Main
> (Main.java:startMesosDriver(119)) - starting mesosDriver..
>
> 2016-05-23 03:38:21,534 INFO [main] scheduler.MyriadDriverManager
> (MyriadDriverManager.java:startDriver(51)) - Starting driver...
>
> 2016-05-23 03:38:21,534 INFO [main] scheduler.MyriadDriver
> (MyriadDriver.java:start(49)) - Starting driver
>
> 2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@712: Client
> environment:zookeeper.version=zookeeper C client 3.4.5
>
> 2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@716: Client
> environment:host.name=qa101-139
>
> 2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@723: Client
> environment:os.name=Linux
>
> 2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@724: Client
> environment:os.arch=3.13.0-57-generic
>
> 2016-05-23 03:38:21,535:7(0x7f4e9b3ac700):ZOO_INFO@log_env@725: Client
> environment:os.version=#95-Ubuntu SMP Fri Jun 19 09:28:15 UTC 2015
>
> I0523 03:38:21.535646 24 sched.cpp:222] Version: 0.28.1
>
> 2016-05-23 03:38:21,535 INFO [main] scheduler.MyriadDriver
> (MyriadDriver.java:start(51)) - Driver started with status: DRIVER_RUNNING
>
> 2016-05-23 03:38:21,536 INFO [main] scheduler.MyriadDriverManager
> (MyriadDriverManager.java:startDriver(53)) - Driver started with status:
> DRIVER_RUNNING
>
> 2016-05-23 03:38:21,536 INFO [main] myriad.Main
> (Main.java:startMesosDriver(121)) - started mesosDriver..
>
> 2016-05-23 03:38:21,536:7(0x7f4e8d685700):ZOO_INFO@check_events@1703:
> initiated connection to server [10.10.101.139:5181]
>
> 2016-05-23 03:38:21,536 INFO [main] interceptor.CompositeInterceptor
> (CompositeInterceptor.java:register(74)) - Registered
> org.apache.myriad.policy.LeastAMNodesFirstPolicy into the registry.
>
> 2016-05-23 03:38:21,539 INFO [main] myriad.Main
> (Main.java:startNMInstances(226)) - Launching 1 NM(s) with profile medium
>
> 2016-05-23 03:38:21,540 INFO [main] scheduler.MyriadOperations
> (MyriadOperations.java:flexUpCluster(80)) - Adding 1 NM instances to cluster
>
> 2016-05-23 03:38:21,555:7(0x7f4e8d685700):ZOO_INFO@check_events@1750:
> session establishment complete on server [10.10.101.139:5181],
> sessionId=0x15314ddb816d02b, negotiated timeout=10000
>
> I0523 03:38:21.555871 97 group.cpp:349] Group process (group(1)@
> 10.10.101.139:57196) connected to ZooKeeper
>
> I0523 03:38:21.555945 97 group.cpp:831] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
>
> I0523 03:38:21.555979 97 group.cpp:427] Trying to create path '/mesos'
> in ZooKeeper
>
> I0523 03:38:21.557073 98 detector.cpp:152] Detected a new leader:
> (id='1')
>
> I0523 03:38:21.557235 75 group.cpp:700] Trying to get
> '/mesos/json.info_0000000001' in ZooKeeper
>
> I0523 03:38:21.558002 97 detector.cpp:479] A new leading master (UPID=
> master@10.10.101.139:5050) is detected
>
> I0523 03:38:21.558116 76 sched.cpp:326] New master detected at
> master@10.10.101.139:5050
>
> I0523 03:38:21.558442 76 sched.cpp:336] No credentials provided.
> Attempting to register without authentication
>
> I0523 03:38:21.559672 93 sched.cpp:703] Framework registered with
> 8114e114-db5f-4faa-afb3-ba1ae29e6368-0023
>
> 2016-05-23 03:38:21,630 INFO [pool-2-thread-1]
> handlers.RegisteredEventHandler (RegisteredEventHandler.java:onEvent(41)) -
> Received event: org.apache.myriad.scheduler.event.RegisteredEvent@27492832
> with frameworkId: value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-0023"
>
> 2016-05-23 03:38:21,691 INFO [main] state.SchedulerState
> (SchedulerState.java:addNodes(77)) - Marked taskId
> nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 pending, size of pending
> queue for nm is: 0
>
> 2016-05-23 03:38:21,702 INFO [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:onEvent(100)) - Received offers 1
>
> 2016-05-23 03:38:21,702 INFO [main] scheduler.MyriadOperations
> (MyriadOperations.java:flexUpAService(139)) - Adding 1 jobhistory instances
> to cluster
>
> 2016-05-23 03:38:21,708 INFO [main] state.SchedulerState
> (SchedulerState.java:addNodes(77)) - Marked taskId
> jobhistory.jobhistory.78b0a6a2-a47d-412f-869d-6fd9872ef85a pending, size of
> pending queue for jobhistory is: 0
>
> 2016-05-23 03:38:21,713 INFO [main]
> interceptor.MyriadInitializationInterceptor
> (MyriadInitializationInterceptor.java:init(54)) - Initialized myriad.
>
> 2016-05-23 03:38:21,714 WARN [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:matches(198)) - Ignoring unknown resource
> type: dfsio_spindles
>
> 2016-05-23 03:38:21,724 INFO [pool-2-thread-3]
> scheduler.TaskFactory$NMTaskFactoryImpl
> (TaskFactory.java:getCommandInfo(138)) - Getting Hadoop distribution from:
> http://172.17.0.2:8192/api/config.tgz
>
> 2016-05-23 03:38:21,749 INFO [main] ipc.CallQueueManager
> (CallQueueManager.java:<init>(53)) - Using callQueue class
> java.util.concurrent.LinkedBlockingQueue
>
> 2016-05-23 03:38:21,758 INFO [Socket Reader #1 for port 8031] ipc.Server
> (Server.java:run(606)) - Starting Socket Reader #1 for port 8031
>
> 2016-05-23 03:38:21,774 INFO [main] pb.RpcServerFactoryPBImpl
> (RpcServerFactoryPBImpl.java:createServer(174)) - Adding protocol
> org.apache.hadoop.yarn.server.api.ResourceTrackerPB to the server
>
> 2016-05-23 03:38:21,775 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(836)) - IPC Server Responder: starting
>
> 2016-05-23 03:38:21,775 INFO [IPC Server listener on 8031] ipc.Server
> (Server.java:run(676)) - IPC Server listener on 8031: starting
>
> 2016-05-23 03:38:21,786 INFO [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:onEvent(139)) - Launching task:
> nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 using offer: value:
> "8114e114-db5f-4faa-afb3-ba1ae29e6368-O98094"
>
>
> 2016-05-23 03:38:21,802 INFO [main] ipc.CallQueueManager
> (CallQueueManager.java:<init>(53)) - Using callQueue class
> java.util.concurrent.LinkedBlockingQueue
>
> 2016-05-23 03:38:21,803 INFO [pool-2-thread-9]
> handlers.ExecutorLostEventHandler
> (ExecutorLostEventHandler.java:onEvent(39)) - Executor value:
> "myriad_executor8114e114-db5f-4faa-afb3-ba1ae29e6368-00238114e114-db5f-4faa-afb3-ba1ae29e6368-O980948114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
>
> of slave value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
>
> lost with exit status: -1
>
> 2016-05-23 03:38:21,809 INFO [Socket Reader #1 for port 8030] ipc.Server
> (Server.java:run(606)) - Starting Socket Reader #1 for port 8030
>
> 2016-05-23 03:38:21,824 INFO [pool-2-thread-5]
> handlers.StatusUpdateEventHandler
> (StatusUpdateEventHandler.java:onEvent(60)) - Status Update for task:
> nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 | state: TASK_FAILED
>
> 2016-05-23 03:38:21,852 INFO [main] pb.RpcServerFactoryPBImpl
> (RpcServerFactoryPBImpl.java:createServer(174)) - Adding protocol
> org.apache.hadoop.yarn.api.ApplicationMasterProtocolPB to the server
>
> 2016-05-23 03:38:21,853 INFO [IPC Server Responder] ipc.Server
> (Server.java:run(836)) - IPC Server Responder: starting
>
> 2016-05-23 03:38:21,853 INFO [IPC Server listener on 8030] ipc.Server
> (Server.java:run(676)) - IPC Server listener on 8030: starting
>
> 2016-05-23 03:38:22,504 INFO [IPC Server listener on 8033] ipc.Server
> (Server.java:run(676)) - IPC Server listener on 8033: starting
>
> 2016-05-23 03:38:22,755 INFO [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:onEvent(100)) - Received offers 1
>
> 2016-05-23 03:38:22,755 WARN [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:matches(198)) - Ignoring unknown resource
> type: dfsio_spindles
>
> 2016-05-23 03:38:22,756 INFO [pool-2-thread-3]
> scheduler.TaskFactory$NMTaskFactoryImpl
> (TaskFactory.java:getCommandInfo(138)) - Getting Hadoop distribution from:
> http://172.17.0.2:8192/api/config.tgz
>
> 2016-05-23 03:38:22,757 INFO [pool-2-thread-3]
> handlers.ResourceOffersEventHandler
> (ResourceOffersEventHandler.java:onEvent(139)) - Launching task:
> nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 using offer: value:
> "8114e114-db5f-4faa-afb3-ba1ae29e6368-O98095"
>
>
> 2016-05-23 03:38:22,768 INFO [pool-2-thread-9]
> handlers.ExecutorLostEventHandler
> (ExecutorLostEventHandler.java:onEvent(39)) - Executor value:
> "myriad_executor8114e114-db5f-4faa-afb3-ba1ae29e6368-00238114e114-db5f-4faa-afb3-ba1ae29e6368-O980958114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
>
> of slave value: "8114e114-db5f-4faa-afb3-ba1ae29e6368-S1"
>
> lost with exit status: -1
>
> 2016-05-23 03:38:22,773 INFO [pool-2-thread-5]
> handlers.StatusUpdateEventHandler
> (StatusUpdateEventHandler.java:onEvent(60)) - Status Update for task:
> nm.medium.ce650d07-e6a6-48bf-aef3-be11c7385bd0 | state: TASK_FAILED
>
> ====================================
>
> To debug above failure, I checked the NM task stdout/stderr but failed to
> get any logs (see screenshot attached)
>
> Then, On attaching to the running containers, found following from the
> docker container:
>
> root@qa101-139:~/myriad/myriad-0.2.0-incubating-rc2/docker# docker exec
> -it 8f730dc2de1a /bin/bash
>
> yarn@qa101-139:/$
>
> yarn@qa101-139:/$ ps -ef
>
> UID PID PPID C STIME TTY TIME CMD
>
> yarn 1 0 0 03:38 ? 00:00:00 /bin/sh -c
> /usr/local/hadoop/bin/yarn resourcemanager
>
> yarn 7 1 14 03:38 ? 00:00:14 /usr/bin/java
> -Dproc_resourcemanager -Xmx1000m -Dhadoop.log.dir=/usr/local/hadoop/logs
> -Dyarn.log.dir=/usr/l
>
> yarn 296 0 0 03:39 ? 00:00:00 /bin/bash
>
> yarn 302 0 0 03:39 ? 00:00:00 /bin/bash
>
> yarn 309 302 0 03:39 ? 00:00:00 ps -ef
>
> yarn@qa101-139:/$ ls -l /usr/local/hadoop/
>
> total 52
>
> -rw-r--r-- 1 root root 15429 Apr 10 2015 LICENSE.txt
>
> -rw-r--r-- 1 root root 101 Apr 10 2015 NOTICE.txt
>
> -rw-r--r-- 1 root root 1366 Apr 10 2015 README.txt
>
> drwxr-xr-x 2 root root 4096 May 22 19:22 bin
>
> drwxr-xr-x 6 root root 4096 May 22 20:42 etc
>
> drwxr-xr-x 2 root root 4096 May 22 19:22 include
>
> drwxr-xr-x 4 root root 4096 May 22 19:22 lib
>
> drwxr-xr-x 2 root root 4096 May 22 19:22 libexec
>
> drwxr-xr-x 2 root root 4096 May 22 19:22 sbin
>
> drwxr-xr-x 7 root root 4096 May 22 20:42 share
>
> yarn@qa101-139:/$ ls -l /usr/local/hadoop/etc/hadoop/
>
> total 16
>
> -rw-r--r-- 1 root root 1340 May 22 20:02 mapred-site.xml
>
> -rw-r--r-- 1 root root 3395 May 23 03:38 myriad-config-default.yml
>
> -rw-r--r-- 1 root root 4207 May 23 00:27 yarn-site.xml
>
> yarn@qa101-139:/$ cat
> /usr/local/hadoop/etc/hadoop/myriad-config-default.yml
>
> mesosMaster: zk://10.10.101.139:5181/mesos ->> (Running on the host
> outside of the container)
>
> #Container information for the node managers
>
> containerInfo:
>
> type: DOCKER
>
> dockerInfo:
>
> image: sarjeet/myriad
>
> volumes:
>
> -
>
> containerPath: /tmp
>
> hostPath: /tmp
>
> checkpoint: false
>
> frameworkFailoverTimeout: 43200000
>
> frameworkName: MyriadAlpha
>
> frameworkRole:
>
> frameworkUser: mapr
>
> # running the resource manager.
>
> frameworkSuperUser: root # To be depricated, currently permissions need
> set by a superuser due to Mesos-1790. Must be
>
> # root or have passwordless sudo. Required if
> nodeManagerURI set, ignored otherwise.
>
> nativeLibrary: /usr/local/lib/libmesos.so
>
> zkServers: 10.10.101.139:5181
>
> zkTimeout: 20000
>
> restApiPort: 8192
>
> profiles:
>
> zero: # NMs launched with this profile dynamically obtain cpu/mem from
> Mesos
>
> cpu: 0
>
> mem: 0
>
> spindles: 0
>
> small:
>
> cpu: 2
>
> mem: 2048
>
> spindles: 1
>
> medium:
>
> cpu: 4
>
> mem: 4096
>
> spindles: 2
>
> large:
>
> cpu: 10
>
> mem: 12288
>
> spindles: 4
>
> nmInstances: # NMs to start with. Requires at least 1 NM with a non-zero
> profile.
>
> medium: 1 # <profile_name : instances>
>
> rebalancer: false
>
> haEnabled: true
>
> servedConfigPath: /dist/config.tgz
>
> nodemanager:
>
> jvmMaxMemoryMB: 1024
>
> cpus: 0.2
>
> cgroups: true
>
> executor:
>
> jvmMaxMemoryMB: 256
>
> configUri: http://172.17.0.1:8192/api/config.tgz
>
> path:
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.1.0.jar
>
> #The following should be used for a remotely distributed URI, hdfs
> assumed but other URI types valid.
>
> #nodeManagerUri: hdfs://namenode:port/dist/hadoop-2.7.0.tar.gz
>
> #path:
> file:///opt/mapr/myriad/myriad-0.1/lib/myriad-executor-runnable-0.1.0.jar
>
> yarnEnvironment:
>
> YARN_NODEMANAGER_OPTS: -Dcluster.name.prefix=/cluster1
> -Dnodemanager.resource.io-spindles=4.0
>
> YARN_HOME: /opt/mapr/hadoop/hadoop-2.7.0
>
> HADOOP_CONF_DIR: /mnt/mesos/sandbox/config
>
> HADOOP_TMP_DIR: /tmp
>
> HADOOP_LOG_DIR: /mnt/mesos/sandbox
>
> #JAVA_HOME: /usr/lib/jvm/java-default #System dependent, but sometimes
> necessary
>
> mesosAuthenticationPrincipal:
>
> mesosAuthenticationSecretFilename:
>
> yarn@qa101-139:/$ netstat -anlp | grep 8088
>
> tcp6 0 0 10.10.101.139:8088 :::*
> LISTEN 7/java
>
> yarn@qa101-139:/$ netstat -anlp | grep 8192
>
> tcp6 0 0 :::8192 :::*
> LISTEN 7/java
>
> yarn@qa101-139:/$
>
> ====================================
>
> Reference: https://github.com/apache/incubator-myriad/tree/master/docker
>
> I might not have looked deeper enough to see if there was any
> configuration issue on launching docker RM, but in case, there is a trivial
> fix or config I missed, I can give this another try. Let me know if there
> was anything I missed?
> - Sarjeet Singh
>