You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mesos.apache.org by "Tim Nolet (JIRA)" <ji...@apache.org> on 2014/08/27 19:10:00 UTC
[jira] [Comment Edited] (MESOS-1737) Isolation=external result in
core dump on 0.20.0
[ https://issues.apache.org/jira/browse/MESOS-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14112456#comment-14112456 ]
Tim Nolet edited comment on MESOS-1737 at 8/27/14 5:09 PM:
-----------------------------------------------------------
Sure, no problem. It complains about a missing file or directory. This is weird. All dirs and files that are mentioned in the arguments exist. Also, the mesos slave starts fine when I just leave out the "--containerizers and --containerizer_path" arguments. See both scenarios below:
*WITH containerizer arguments: stack trace:*
{{
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos --containerizers=external --containerizer_path=/usr/local/bin/deimos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 206)]
[New Thread 0x7ffff4555700 (LWP 207)]
[New Thread 0x7ffff3d54700 (LWP 208)]
[New Thread 0x7ffff3553700 (LWP 209)]
[New Thread 0x7ffff2d52700 (LWP 210)]
[New Thread 0x7ffff2551700 (LWP 211)]
[New Thread 0x7ffff1d50700 (LWP 212)]
[New Thread 0x7ffff154f700 (LWP 213)]
[New Thread 0x7ffff0d4e700 (LWP 214)]
I0827 17:01:39.182112 202 logging.cpp:142] INFO level logging started!
I0827 17:01:39.183714 202 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:01:39.184777 202 main.cpp:128] Version: 0.20.0
I0827 17:01:39.185250 202 main.cpp:131] Git tag: 0.20.0
I0827 17:01:39.185858 202 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
Program received signal SIGSEGV, Segmentation fault.
_int_malloc (av=0x7ffff5956760 <main_arena>, bytes=26) at malloc.c:3302
3302 malloc.c: No such file or directory.
(gdb) Quit
}}
*WITHOUT containerizer arguments: all good*
{{
(gdb) run
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 269)]
[New Thread 0x7ffff4555700 (LWP 270)]
[New Thread 0x7ffff3d54700 (LWP 271)]
[New Thread 0x7ffff3553700 (LWP 272)]
[New Thread 0x7ffff2d52700 (LWP 273)]
[New Thread 0x7ffff2551700 (LWP 274)]
[New Thread 0x7ffff1d50700 (LWP 275)]
[New Thread 0x7ffff154f700 (LWP 276)]
[New Thread 0x7ffff0d4e700 (LWP 277)]
I0827 17:05:50.842244 265 logging.cpp:142] INFO level logging started!
I0827 17:05:50.843018 265 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:05:50.844144 265 main.cpp:128] Version: 0.20.0
I0827 17:05:50.844662 265 main.cpp:131] Git tag: 0.20.0
I0827 17:05:50.846150 265 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
I0827 17:05:50.846617 265 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem
I0827 17:05:50.847182 265 main.cpp:149] Starting Mesos slave
2014-08-27 17:05:50,847:265(0x7ffff2551700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-27 17:05:50,848:265(0x7ffff2551700):ZOO_INFO@log_env@716: Client environment:host.name=f321e9ca8261
2014-08-27 17:05:50,849:265(0x7ffff2551700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-08-27 17:05:50,850:265(0x7ffff2551700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.8+
2014-08-27 17:05:50,851:265(0x7ffff2551700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Fri Aug 15 22:29:31 UTC 2014
2014-08-27 17:05:50,852:265(0x7ffff2551700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
I0827 17:05:50.859122 265 slave.cpp:167] Slave started on 1)@10.1.0.36:5051
I0827 17:05:50.864536 265 slave.cpp:278] Slave resources: cpus(*):1; mem(*):499; disk(*):11280; ports(*):[31000-32000]
2014-08-27 17:05:50,889:265(0x7ffff2551700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7ffff6caea30 sessionId=0 sessionPasswd=<null> context=0x7fffec001f30 flags=0
I0827 17:05:50.894484 265 slave.cpp:306] Slave hostname: f321e9ca8261
I0827 17:05:50.895529 265 slave.cpp:307] Slave checkpoint: true
I0827 17:05:50.900344 269 state.cpp:33] Recovering state from '/var/lib/mesos/meta'
I0827 17:05:50.901289 269 state.cpp:62] Failed to find the latest slave from '/var/lib/mesos/meta'
I0827 17:05:50.907994 272 status_update_manager.cpp:193] Recovering status update manager
I0827 17:05:50.909273 273 containerizer.cpp:252] Recovering containerizer
I0827 17:05:50.914098 265 slave.cpp:3195] Finished recovery
[New Thread 0x7fffdbbe7700 (LWP 278)]
[New Thread 0x7fffdb3e6700 (LWP 279)]
2014-08-27 17:05:50,922:265(0x7fffdbbe7700):ZOO_INFO@check_events@1703: initiated connection to server [::1:2181]
2014-08-27 17:05:50,925:265(0x7fffdbbe7700):ZOO_INFO@check_events@1750: session establishment complete on server [::1:2181], sessionId=0x148186e1c770004, negotiated timeout=10000
I0827 17:05:50.926491 275 group.cpp:313] Group process (group(1)@10.1.0.36:5051) connected to ZooKeeper
I0827 17:05:50.926971 275 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0827 17:05:50.927428 275 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0827 17:05:50.940911 275 detector.cpp:138] Detected a new leader: (id='0')
I0827 17:05:50.941494 275 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper
I0827 17:05:50.951995 275 detector.cpp:426] A new leading master (UPID=master@10.1.0.36:5050) is detected
I0827 17:05:50.953053 275 slave.cpp:589] New master detected at master@10.1.0.36:5050
I0827 17:05:50.954869 275 slave.cpp:625] No credentials provided. Attempting to register without authentication
I0827 17:05:50.955654 275 slave.cpp:636] Detecting new master
I0827 17:05:50.955529 269 status_update_manager.cpp:167] New master detected at master@10.1.0.36:5050
I0827 17:05:51.957218 275 slave.cpp:754] Registered with master master@10.1.0.36:5050; given slave ID 20140827-155409-603980042-5050-10-0
I0827 17:05:51.957676 275 slave.cpp:767] Checkpointing SlaveInfo to '/var/lib/mesos/meta/slaves/20140827-155409-603980042-5050-10-0/slave.info'
...
}}
was (Author: tnolet):
Sure, no problem. It complains about a missing file or directory. This is weird. All dirs and files that are mentioned in the arguments exist. Also, the mesos slave starts fine when I just leave out the "--containerizers and --containerizer_path" arguments. See both scenarios below:
### WITH containerizer arguments: stack trace ###
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos --containerizers=external --containerizer_path=/usr/local/bin/deimos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 206)]
[New Thread 0x7ffff4555700 (LWP 207)]
[New Thread 0x7ffff3d54700 (LWP 208)]
[New Thread 0x7ffff3553700 (LWP 209)]
[New Thread 0x7ffff2d52700 (LWP 210)]
[New Thread 0x7ffff2551700 (LWP 211)]
[New Thread 0x7ffff1d50700 (LWP 212)]
[New Thread 0x7ffff154f700 (LWP 213)]
[New Thread 0x7ffff0d4e700 (LWP 214)]
I0827 17:01:39.182112 202 logging.cpp:142] INFO level logging started!
I0827 17:01:39.183714 202 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:01:39.184777 202 main.cpp:128] Version: 0.20.0
I0827 17:01:39.185250 202 main.cpp:131] Git tag: 0.20.0
I0827 17:01:39.185858 202 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
Program received signal SIGSEGV, Segmentation fault.
_int_malloc (av=0x7ffff5956760 <main_arena>, bytes=26) at malloc.c:3302
3302 malloc.c: No such file or directory.
(gdb) Quit
### WITHOUT containerizer arguments: all good ###
(gdb) run
Starting program: /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=10.1.0.36 --work_dir=/var/lib/mesos
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff4d56700 (LWP 269)]
[New Thread 0x7ffff4555700 (LWP 270)]
[New Thread 0x7ffff3d54700 (LWP 271)]
[New Thread 0x7ffff3553700 (LWP 272)]
[New Thread 0x7ffff2d52700 (LWP 273)]
[New Thread 0x7ffff2551700 (LWP 274)]
[New Thread 0x7ffff1d50700 (LWP 275)]
[New Thread 0x7ffff154f700 (LWP 276)]
[New Thread 0x7ffff0d4e700 (LWP 277)]
I0827 17:05:50.842244 265 logging.cpp:142] INFO level logging started!
I0827 17:05:50.843018 265 main.cpp:126] Build: 2014-08-22 05:05:59 by root
I0827 17:05:50.844144 265 main.cpp:128] Version: 0.20.0
I0827 17:05:50.844662 265 main.cpp:131] Git tag: 0.20.0
I0827 17:05:50.846150 265 main.cpp:135] Git SHA: f421ffdf8d32a8834b3a6ee483b5b59f65956497
I0827 17:05:50.846617 265 containerizer.cpp:89] Using isolation: posix/cpu,posix/mem
I0827 17:05:50.847182 265 main.cpp:149] Starting Mesos slave
2014-08-27 17:05:50,847:265(0x7ffff2551700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2014-08-27 17:05:50,848:265(0x7ffff2551700):ZOO_INFO@log_env@716: Client environment:host.name=f321e9ca8261
2014-08-27 17:05:50,849:265(0x7ffff2551700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2014-08-27 17:05:50,850:265(0x7ffff2551700):ZOO_INFO@log_env@724: Client environment:os.arch=3.15.8+
2014-08-27 17:05:50,851:265(0x7ffff2551700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Fri Aug 15 22:29:31 UTC 2014
2014-08-27 17:05:50,852:265(0x7ffff2551700):ZOO_INFO@log_env@733: Client environment:user.name=(null)
I0827 17:05:50.859122 265 slave.cpp:167] Slave started on 1)@10.1.0.36:5051
I0827 17:05:50.864536 265 slave.cpp:278] Slave resources: cpus(*):1; mem(*):499; disk(*):11280; ports(*):[31000-32000]
2014-08-27 17:05:50,889:265(0x7ffff2551700):ZOO_INFO@log_env@741: Client environment:user.home=/root
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@log_env@753: Client environment:user.dir=/
2014-08-27 17:05:50,890:265(0x7ffff2551700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=localhost:2181 sessionTimeout=10000 watcher=0x7ffff6caea30 sessionId=0 sessionPasswd=<null> context=0x7fffec001f30 flags=0
I0827 17:05:50.894484 265 slave.cpp:306] Slave hostname: f321e9ca8261
I0827 17:05:50.895529 265 slave.cpp:307] Slave checkpoint: true
I0827 17:05:50.900344 269 state.cpp:33] Recovering state from '/var/lib/mesos/meta'
I0827 17:05:50.901289 269 state.cpp:62] Failed to find the latest slave from '/var/lib/mesos/meta'
I0827 17:05:50.907994 272 status_update_manager.cpp:193] Recovering status update manager
I0827 17:05:50.909273 273 containerizer.cpp:252] Recovering containerizer
I0827 17:05:50.914098 265 slave.cpp:3195] Finished recovery
[New Thread 0x7fffdbbe7700 (LWP 278)]
[New Thread 0x7fffdb3e6700 (LWP 279)]
2014-08-27 17:05:50,922:265(0x7fffdbbe7700):ZOO_INFO@check_events@1703: initiated connection to server [::1:2181]
2014-08-27 17:05:50,925:265(0x7fffdbbe7700):ZOO_INFO@check_events@1750: session establishment complete on server [::1:2181], sessionId=0x148186e1c770004, negotiated timeout=10000
I0827 17:05:50.926491 275 group.cpp:313] Group process (group(1)@10.1.0.36:5051) connected to ZooKeeper
I0827 17:05:50.926971 275 group.cpp:787] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0827 17:05:50.927428 275 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0827 17:05:50.940911 275 detector.cpp:138] Detected a new leader: (id='0')
I0827 17:05:50.941494 275 group.cpp:658] Trying to get '/mesos/info_0000000000' in ZooKeeper
I0827 17:05:50.951995 275 detector.cpp:426] A new leading master (UPID=master@10.1.0.36:5050) is detected
I0827 17:05:50.953053 275 slave.cpp:589] New master detected at master@10.1.0.36:5050
I0827 17:05:50.954869 275 slave.cpp:625] No credentials provided. Attempting to register without authentication
I0827 17:05:50.955654 275 slave.cpp:636] Detecting new master
I0827 17:05:50.955529 269 status_update_manager.cpp:167] New master detected at master@10.1.0.36:5050
I0827 17:05:51.957218 275 slave.cpp:754] Registered with master master@10.1.0.36:5050; given slave ID 20140827-155409-603980042-5050-10-0
I0827 17:05:51.957676 275 slave.cpp:767] Checkpointing SlaveInfo to '/var/lib/mesos/meta/slaves/20140827-155409-603980042-5050-10-0/slave.info'
> Isolation=external result in core dump on 0.20.0
> ------------------------------------------------
>
> Key: MESOS-1737
> URL: https://issues.apache.org/jira/browse/MESOS-1737
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Affects Versions: 0.20.0
> Reporter: Tim Nolet
> Assignee: Timothy Chen
>
> When upgrading from 0.19.1 to 0.20.0, any slaves started with the standard deimos setup fail hard on startup. The following command spits out about 20.000 errors before core dumping:
> /etc/mesos-slave# /usr/local/sbin/mesos-slave --master=zk://localhost:2181/mesos --port=5051 --log_dir=/var/log/mesos --ip=172.17.8.101 --work_dir=/var/lib/mesos --isolation=external --containerizer_path=/usr/local/bin/deimos
> output:
> ....
> W0827 15:20:18.366271 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366580 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366631 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366683 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366714 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> W0827 15:20:18.366752 721 containerizer.cpp:159] The 'external' isolation flag is deprecated, please update your flags to '--containerizers=external'.
> Segmentation fault (core dumped)
--
This message was sent by Atlassian JIRA
(v6.2#6252)