You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Tim Nolet (JIRA)" <ji...@apache.org> on 2014/08/27 19:10:00 UTC

[jira] [Comment Edited] (MESOS-1737) Isolation=external result in core dump on 0.20.0

    [ https://issues.apache.org/jira/browse/MESOS-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112456#comment-14112456 ] 

Tim Nolet edited comment on MESOS-1737 at 8/27/14 5:09 PM:
-----------------------------------------------------------

Sure, no problem. It complains about a missing file or directory. This is weird. All dirs and files that are mentioned in the arguments exist. Also, the mesos slave starts fine when I just leave out the "--containerizers and --containerizer_path" arguments. See both scenarios below:

*WITH containerizer arguments: stack trace:*
{{
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos --containerizers=external --containerizer_path=/usr/local/bin/deimos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 206)]
[New Thread 0x7ffff4555700 (LWP 207)]
[New Thread 0x7ffff3d54700 (LWP 208)]
[New Thread 0x7ffff3553700 (LWP 209)]
[New Thread 0x7ffff2d52700 (LWP 210)]
[New Thread 0x7ffff2551700 (LWP 211)]
[New Thread 0x7ffff1d50700 (LWP 212)]
[New Thread 0x7ffff154f700 (LWP 213)]
[New Thread 0x7ffff0d4e700 (LWP 214)]
I0827 17:01:39.182112   202 logging.cpp:142] INFO level logging started!
I0827 17:01:39.183714   202 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:01:39.184777   202 main.cpp:128] Version: 0.20.0
I0827 17:01:39.185250   202 main.cpp:131] Git tag: 0.20.0
I0827 17:01:39.185858   202 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497

Program received signal SIGSEGV, Segmentation fault.
_int_malloc (av=0x7ffff5956760 <main_arena>, bytes=26) at malloc.c:3302
3302    malloc.c: No such file or directory.
(gdb) Quit
}}

*WITHOUT containerizer arguments: all good*
{{
(gdb) run
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 269)]
[New Thread 0x7ffff4555700 (LWP 270)]
[New Thread 0x7ffff3d54700 (LWP 271)]
[New Thread 0x7ffff3553700 (LWP 272)]
[New Thread 0x7ffff2d52700 (LWP 273)]
[New Thread 0x7ffff2551700 (LWP 274)]
[New Thread 0x7ffff1d50700 (LWP 275)]
[New Thread 0x7ffff154f700 (LWP 276)]
[New Thread 0x7ffff0d4e700 (LWP 277)]
I0827 17:05:50.842244   265 logging.cpp:142] INFO level logging started!
I0827 17:05:50.843018   265 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:05:50.844144   265 main.cpp:128] Version: 0.20.0
I0827 17:05:50.844662   265 main.cpp:131] Git tag: 0.20.0
I0827 17:05:50.846150   265 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
I0827 17:05:50.846617   265 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem
I0827 17:05:50.847182   265 main.cpp:149] Starting Mesos slave
2014-08-27 17:05:50,847:265(0x7ffff2551700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-27 17:05:50,848:265(0x7ffff2551700):ZOO_INFO@log_env@716: Client environment:host.name=f321e9ca8261
2014-08-27 17:05:50,849:265(0x7ffff2551700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-08-27 17:05:50,850:265(0x7ffff2551700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.8+
2014-08-27 17:05:50,851:265(0x7ffff2551700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Fri Aug 15 22:29:31 UTC 2014
2014-08-27 17:05:50,852:265(0x7ffff2551700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
I0827 17:05:50.859122   265 slave.cpp:167] Slave started on 1)@10.1.0.36:5051
I0827 17:05:50.864536   265 slave.cpp:278] Slave resources: cpus(*):1; mem(*):499; disk(*):11280; ports(*):[31000-32000]
2014-08-27 17:05:50,889:265(0x7ffff2551700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7ffff6caea30 sessionId=0 sessionPasswd=<null> context=0x7fffec001f30 flags=0
I0827 17:05:50.894484   265 slave.cpp:306] Slave hostname: f321e9ca8261
I0827 17:05:50.895529   265 slave.cpp:307] Slave checkpoint: true
I0827 17:05:50.900344   269 state.cpp:33] Recovering state from '/var/lib/mesos/meta'
I0827 17:05:50.901289   269 state.cpp:62] Failed to find the latest slave from '/var/lib/mesos/meta'
I0827 17:05:50.907994   272 status_update_manager.cpp:193] Recovering status update manager
I0827 17:05:50.909273   273 containerizer.cpp:252] Recovering containerizer
I0827 17:05:50.914098   265 slave.cpp:3195] Finished recovery
[New Thread 0x7fffdbbe7700 (LWP 278)]
[New Thread 0x7fffdb3e6700 (LWP 279)]
2014-08-27 17:05:50,922:265(0x7fffdbbe7700):ZOO_INFO@check_events@1703: initiated connection to server [::1:2181]
2014-08-27 17:05:50,925:265(0x7fffdbbe7700):ZOO_INFO@check_events@1750: session establishment complete on server [::1:2181], sessionId=0x148186e1c770004, negotiated timeout=10000
I0827 17:05:50.926491   275 group.cpp:313] Group process (group(1)@10.1.0.36:5051) connected to ZooKeeper
I0827 17:05:50.926971   275 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0827 17:05:50.927428   275 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0827 17:05:50.940911   275 detector.cpp:138] Detected a new leader: (id='0')
I0827 17:05:50.941494   275 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper
I0827 17:05:50.951995   275 detector.cpp:426] A new leading master (UPID=master@10.1.0.36:5050) is detected
I0827 17:05:50.953053   275 slave.cpp:589] New master detected at master@10.1.0.36:5050
I0827 17:05:50.954869   275 slave.cpp:625] No credentials provided. Attempting to register without authentication
I0827 17:05:50.955654   275 slave.cpp:636] Detecting new master
I0827 17:05:50.955529   269 status_update_manager.cpp:167] New master detected at master@10.1.0.36:5050
I0827 17:05:51.957218   275 slave.cpp:754] Registered with master master@10.1.0.36:5050; given slave ID 20140827-155409-603980042-5050-10-0
I0827 17:05:51.957676   275 slave.cpp:767] Checkpointing SlaveInfo to '/var/lib/mesos/meta/slaves/20140827-155409-603980042-5050-10-0/slave.info'
...
}}



was (Author: tnolet):
Sure, no problem. It complains about a missing file or directory. This is weird. All dirs and files that are mentioned in the arguments exist. Also, the mesos slave starts fine when I just leave out the "--containerizers and --containerizer_path" arguments. See both scenarios below:

### WITH containerizer arguments: stack trace ###
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos --containerizers=external --containerizer_path=/usr/local/bin/deimos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 206)]
[New Thread 0x7ffff4555700 (LWP 207)]
[New Thread 0x7ffff3d54700 (LWP 208)]
[New Thread 0x7ffff3553700 (LWP 209)]
[New Thread 0x7ffff2d52700 (LWP 210)]
[New Thread 0x7ffff2551700 (LWP 211)]
[New Thread 0x7ffff1d50700 (LWP 212)]
[New Thread 0x7ffff154f700 (LWP 213)]
[New Thread 0x7ffff0d4e700 (LWP 214)]
I0827 17:01:39.182112   202 logging.cpp:142] INFO level logging started!
I0827 17:01:39.183714   202 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:01:39.184777   202 main.cpp:128] Version: 0.20.0
I0827 17:01:39.185250   202 main.cpp:131] Git tag: 0.20.0
I0827 17:01:39.185858   202 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497

Program received signal SIGSEGV, Segmentation fault.
_int_malloc (av=0x7ffff5956760 <main_arena>, bytes=26) at malloc.c:3302
3302    malloc.c: No such file or directory.
(gdb) Quit


### WITHOUT containerizer arguments: all good ###

(gdb) run
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 269)]
[New Thread 0x7ffff4555700 (LWP 270)]
[New Thread 0x7ffff3d54700 (LWP 271)]
[New Thread 0x7ffff3553700 (LWP 272)]
[New Thread 0x7ffff2d52700 (LWP 273)]
[New Thread 0x7ffff2551700 (LWP 274)]
[New Thread 0x7ffff1d50700 (LWP 275)]
[New Thread 0x7ffff154f700 (LWP 276)]
[New Thread 0x7ffff0d4e700 (LWP 277)]
I0827 17:05:50.842244   265 logging.cpp:142] INFO level logging started!
I0827 17:05:50.843018   265 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:05:50.844144   265 main.cpp:128] Version: 0.20.0
I0827 17:05:50.844662   265 main.cpp:131] Git tag: 0.20.0
I0827 17:05:50.846150   265 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
I0827 17:05:50.846617   265 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem
I0827 17:05:50.847182   265 main.cpp:149] Starting Mesos slave
2014-08-27 17:05:50,847:265(0x7ffff2551700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-27 17:05:50,848:265(0x7ffff2551700):ZOO_INFO@log_env@716: Client environment:host.name=f321e9ca8261
2014-08-27 17:05:50,849:265(0x7ffff2551700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-08-27 17:05:50,850:265(0x7ffff2551700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.8+
2014-08-27 17:05:50,851:265(0x7ffff2551700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Fri Aug 15 22:29:31 UTC 2014
2014-08-27 17:05:50,852:265(0x7ffff2551700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
I0827 17:05:50.859122   265 slave.cpp:167] Slave started on 1)@10.1.0.36:5051
I0827 17:05:50.864536   265 slave.cpp:278] Slave resources: cpus(*):1; mem(*):499; disk(*):11280; ports(*):[31000-32000]
2014-08-27 17:05:50,889:265(0x7ffff2551700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7ffff6caea30 sessionId=0 sessionPasswd=<null> context=0x7fffec001f30 flags=0
I0827 17:05:50.894484   265 slave.cpp:306] Slave hostname: f321e9ca8261
I0827 17:05:50.895529   265 slave.cpp:307] Slave checkpoint: true
I0827 17:05:50.900344   269 state.cpp:33] Recovering state from '/var/lib/mesos/meta'
I0827 17:05:50.901289   269 state.cpp:62] Failed to find the latest slave from '/var/lib/mesos/meta'
I0827 17:05:50.907994   272 status_update_manager.cpp:193] Recovering status update manager
I0827 17:05:50.909273   273 containerizer.cpp:252] Recovering containerizer
I0827 17:05:50.914098   265 slave.cpp:3195] Finished recovery
[New Thread 0x7fffdbbe7700 (LWP 278)]
[New Thread 0x7fffdb3e6700 (LWP 279)]
2014-08-27 17:05:50,922:265(0x7fffdbbe7700):ZOO_INFO@check_events@1703: initiated connection to server [::1:2181]
2014-08-27 17:05:50,925:265(0x7fffdbbe7700):ZOO_INFO@check_events@1750: session establishment complete on server [::1:2181], sessionId=0x148186e1c770004, negotiated timeout=10000
I0827 17:05:50.926491   275 group.cpp:313] Group process (group(1)@10.1.0.36:5051) connected to ZooKeeper
I0827 17:05:50.926971   275 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0827 17:05:50.927428   275 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0827 17:05:50.940911   275 detector.cpp:138] Detected a new leader: (id='0')
I0827 17:05:50.941494   275 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper
I0827 17:05:50.951995   275 detector.cpp:426] A new leading master (UPID=master@10.1.0.36:5050) is detected
I0827 17:05:50.953053   275 slave.cpp:589] New master detected at master@10.1.0.36:5050
I0827 17:05:50.954869   275 slave.cpp:625] No credentials provided. Attempting to register without authentication
I0827 17:05:50.955654   275 slave.cpp:636] Detecting new master
I0827 17:05:50.955529   269 status_update_manager.cpp:167] New master detected at master@10.1.0.36:5050
I0827 17:05:51.957218   275 slave.cpp:754] Registered with master master@10.1.0.36:5050; given slave ID 20140827-155409-603980042-5050-10-0
I0827 17:05:51.957676   275 slave.cpp:767] Checkpointing SlaveInfo to '/var/lib/mesos/meta/slaves/20140827-155409-603980042-5050-10-0/slave.info'


> Isolation=external result in core dump on 0.20.0
> ------------------------------------------------
>
>                 Key: MESOS-1737
>                 URL: https://issues.apache.org/jira/browse/MESOS-1737
>             Project: Mesos
>          Issue Type: Bug
>          Components: containerization
>    Affects Versions: 0.20.0
>            Reporter: Tim Nolet
>            Assignee: Timothy Chen
>
> When upgrading from 0.19.1 to 0.20.0, any slaves started with the standard deimos setup fail hard on startup. The following command spits out about 20.000 errors before core dumping:
> /etc/mesos-slave# /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=172.17.8.101 --work_dir=/var/lib/mesos --isolation=external --containerizer_path=/usr/local/bin/deimos
> output:
> ....
> W0827 15:20:18.366271   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366580   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366631   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366683   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366714   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366752   721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> Segmentation fault (core dumped)



--
This message was sent by Atlassian JIRA
(v6.2#6252)