You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by daijia <ji...@intsig.com> on 2014/07/30 03:42:54 UTC
How to submit Pyspark job in mesos?
Dear all,
I have spark1.0.0 and mesos0.18.1. After setting in mesos and spark
and starting the mesos cluster, I try to run the pyspark job by the command
below:
spark-submit /path/to/my_pyspark_job.py --master
mesos://192.168.0.21:5050
It occurs error below:
14/07/29 18:40:49 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/29 18:40:49 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4041
14/07/29 18:40:49 INFO ui.SparkUI: Started SparkUI at http://CentOS-19:4041
14/07/29 18:40:49 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/07/29 18:40:50 INFO scheduler.EventLoggingListener: Logging events to
/tmp/spark-events/my_test.py-1406630449771
14/07/29 18:40:50 INFO util.Utils: Copying
/home/daijia/deal_three_word/my_test.py to
/tmp/spark-4365b01d-b57a-4abb-b39c-cb57b83a28ce/my_test.py
14/07/29 18:40:50 INFO spark.SparkContext: Added file
file:/home/daijia/deal_three_word/my_test.py at
http://192.168.3.91:51188/files/my_test.py with timestamp 1406630450333
I0729 18:40:50.440551 15033 sched.cpp:121] Version: 0.18.1
I0729 18:40:50.442450 15035 sched.cpp:217] New master detected at
master@192.168.3.91:5050
I0729 18:40:50.442570 15035 sched.cpp:225] No credentials provided.
Attempting to register without authentication
I0729 18:40:50.443234 15036 sched.cpp:391] Framework registered with
20140729-174911-1526966464-5050-13758-0006
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Registered as
framework ID 20140729-174911-1526966464-5050-13758-0006
14/07/29 18:40:50 INFO spark.SparkContext: Starting job: count at
/home/daijia/deal_three_word/my_test.py:27
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 0 is
now TASK_LOST
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 1 is
now TASK_LOST
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 3 is
now TASK_LOST
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140729-163345-1526966464-5050-10913-0"
due to too many failures; is Spark installed on it?
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 2 is
now TASK_LOST
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140729-163345-1526966464-5050-10913-2"
due to too many failures; is Spark installed on it?
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Got job 0 (count at
/home/daijia/deal_three_word/my_test.py:27) with 2 output partitions
(allowLocal=false)
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Final stage: Stage 0(count at
/home/daijia/deal_three_word/my_test.py:27)
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Parents of final stage:
List()
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Missing parents: List()
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 4 is
now TASK_LOST
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Submitting Stage 0
(PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 5 is
now TASK_LOST
14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140729-163345-1526966464-5050-10913-1"
due to too many failures; is Spark installed on it?
14/07/29 18:40:50 INFO scheduler.DAGScheduler: Submitting 2 missing tasks
from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
14/07/29 18:40:50 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with
2 tasks
14/07/29 18:41:05 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/07/29 18:41:20 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/07/29 18:41:20 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
It just repeats the last message.
Here is my python scirpt:
#!/usr/bin/env python
#coding=utf-8
from pyspark import SparkContext
sc = SparkContext()
temp = []
for index in range(1000):
temp.append(index)
sc.parallelize(temp).count()
So, the running command is right? Or some other reasons lead to the
problem.
Thanks in advance,
Daijia
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-submit-Pyspark-job-in-mesos-tp10905.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to submit Pyspark job in mesos?
Posted by daijia <ji...@intsig.com>.
I have one master and three slaves.
here is master log:
I0730 16:56:21.126581 23977 main.cpp:127] Build: 2014-07-28 15:34:46 by root
I0730 16:56:21.126745 23977 main.cpp:129] Version: 0.18.1
I0730 16:56:21.127044 23977 main.cpp:178] Starting Mesos master
I0730 16:56:21.127818 23977 master.cpp:240] Master ID:
20140730-165621-1526966464-5050-23977 Hostname: CentOS-19
I0730 16:56:21.127964 23977 master.cpp:322] Master started on
192.168.3.91:5050
I0730 16:56:21.127989 23977 master.cpp:332] Master allowing unauthenticated
frameworks to register!!
I0730 16:56:21.130705 23979 master.cpp:757] The newly elected leader is
master@192.168.3.91:5050 with id 20140730-165621-1526966464-5050-23977
I0730 16:56:21.130739 23979 master.cpp:767] Elected as the leading master!
I0730 16:56:22.628568 23979 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:24.049960 23981 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:26.096403 23982 master.cpp:1933] Attempting to re-register slave
20140730-154530-1526966464-5050-22832-0 at slave(1)@192.168.3.92:5051
(CentOS-20)
I0730 16:56:26.096485 23982 master.cpp:2815] Adding slave
20140730-154530-1526966464-5050-22832-0 at CentOS-20 with cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000]
I0730 16:56:26.096909 23982 hierarchical_allocator_process.hpp:445] Added
slave 20140730-154530-1526966464-5050-22832-0 (CentOS-20) with cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000] (and cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000] available)
I0730 16:56:29.051209 23983 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:29.693975 23984 master.cpp:1933] Attempting to re-register slave
20140730-154530-1526966464-5050-22832-1 at slave(1)@192.168.3.93:5051
(CentOS-21)
I0730 16:56:29.694032 23984 master.cpp:2815] Adding slave
20140730-154530-1526966464-5050-22832-1 at CentOS-21 with mem(*):1000;
cpus(*):4; disk(*):1.00684e+06; ports(*):[31000-32000]
I0730 16:56:29.694634 23984 hierarchical_allocator_process.hpp:445] Added
slave 20140730-154530-1526966464-5050-22832-1 (CentOS-21) with mem(*):1000;
cpus(*):4; disk(*):1.00684e+06; ports(*):[31000-32000] (and mem(*):1000;
cpus(*):4; disk(*):1.00684e+06; ports(*):[31000-32000] available)
I0730 16:56:32.650281 23981 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:35.049878 23980 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:35.189177 23983 master.cpp:1933] Attempting to re-register slave
20140730-154530-1526966464-5050-22832-2 at slave(1)@192.168.3.94:5051
(CentOS-22)
I0730 16:56:35.189231 23983 master.cpp:2815] Adding slave
20140730-154530-1526966464-5050-22832-2 at CentOS-22 with cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000]
I0730 16:56:35.189584 23983 hierarchical_allocator_process.hpp:445] Added
slave 20140730-154530-1526966464-5050-22832-2 (CentOS-22) with cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000] (and cpus(*):4;
mem(*):1000; disk(*):1.00684e+06; ports(*):[31000-32000] available)
I0730 16:56:40.052042 23982 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:42.670194 23978 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:46.049862 23984 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:49.885049 23980 master.cpp:815] Received registration request
from scheduler(1)@192.168.3.91:35068
I0730 16:56:49.885107 23980 master.cpp:833] Registering framework
20140730-165621-1526966464-5050-23977-0000 at
scheduler(1)@192.168.3.91:35068
I0730 16:56:49.885332 23980 hierarchical_allocator_process.hpp:332] Added
framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:56:49.885712 23980 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:50.007832 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-0 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:50.007928 23982 master.hpp:411] Adding task 0 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2
(CentOS-22)
I0730 16:56:50.007961 23982 master.cpp:2451] Launching task 0 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2 (CentOS-22)
I0730 16:56:50.008204 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-1 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:56:50.008251 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:50.008272 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:56:50.008304 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-2 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:56:50.008359 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:50.008374 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:56:50.008433 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:50.008445 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:56:51.054510 23983 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:56:52.693107 23979 http.cpp:391] HTTP request for
'/master/state.json'
W0730 16:56:55.035470 23980 master.cpp:2103] Ignoring unknown exited
executor 0 on slave 20140730-154530-1526966464-5050-22832-2 (CentOS-22)
I0730 16:56:55.036079 23985 master.cpp:2030] Status update TASK_LOST (UUID:
fc590d2c-1be2-4586-be92-bb844756aa3e) for task 0 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.94:5051
I0730 16:56:55.036192 23985 master.hpp:429] Removing task 0 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2
(CentOS-22)
I0730 16:56:55.036350 23985 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: cpus(*):4; mem(*):1000;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-2 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:55.046278 23985 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:55.046386 23985 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:56:55.046886 23985 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:55.051283 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-3 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:56:55.051357 23983 master.hpp:411] Adding task 1 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2
(CentOS-22)
I0730 16:56:55.051398 23983 master.cpp:2451] Launching task 1 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2 (CentOS-22)
I0730 16:56:55.051698 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-4 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:56:55.051771 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:55.051810 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:56:55.051870 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-5 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:56:55.052011 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:55.052047 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:56:55.052255 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:56:55.052289 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:56:57.049111 23984 http.cpp:391] HTTP request for
'/master/state.json'
W0730 16:57:00.038585 23982 master.cpp:2103] Ignoring unknown exited
executor 1 on slave 20140730-154530-1526966464-5050-22832-2 (CentOS-22)
I0730 16:57:00.040513 23978 master.cpp:2030] Status update TASK_LOST (UUID:
35736ef8-fea7-450c-a176-23e38a3a6431) for task 1 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.94:5051
I0730 16:57:00.040618 23978 master.hpp:429] Removing task 1 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-2
(CentOS-22)
I0730 16:57:00.040838 23985 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: cpus(*):4; mem(*):1000;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-2 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:00.043002 23978 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:00.043078 23978 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:00.043900 23978 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:00.050076 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-6 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:00.050292 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:00.050319 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:00.050443 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-7 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:00.050516 23981 master.hpp:411] Adding task 2 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0
(CentOS-20)
I0730 16:57:00.050555 23981 master.cpp:2451] Launching task 2 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0 (CentOS-20)
I0730 16:57:00.050851 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-8 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:00.050860 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:00.050889 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:00.050983 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:00.051000 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:02.050163 23980 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:02.716814 23984 http.cpp:391] HTTP request for
'/master/state.json'
W0730 16:57:05.083808 23985 master.cpp:2103] Ignoring unknown exited
executor 2 on slave 20140730-154530-1526966464-5050-22832-0 (CentOS-20)
I0730 16:57:05.083953 23985 master.cpp:2030] Status update TASK_LOST (UUID:
324e2616-525f-40d0-8216-f01c86535fb6) for task 2 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.92:5051
I0730 16:57:05.084013 23985 master.hpp:429] Removing task 2 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0
(CentOS-20)
I0730 16:57:05.084163 23985 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: cpus(*):4; mem(*):1000;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-0 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.085674 23978 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.085723 23978 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.086299 23978 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.092308 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-9 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.092524 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-10 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:05.092578 23978 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:05.092586 23985 master.hpp:411] Adding task 3 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0
(CentOS-20)
I0730 16:57:05.092603 23978 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:05.092624 23985 master.cpp:2451] Launching task 3 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0 (CentOS-20)
W0730 16:57:05.092892 23980 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:05.092918 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-11 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:05.092944 23980 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:05.093121 23984 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:05.093144 23984 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:08.047802 23985 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:10.164223 23979 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:10.167493 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-12 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:10.167697 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-13 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:10.167701 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:10.167773 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:10.167863 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-14 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:10.167912 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:10.167932 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:10.168016 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:10.168032 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
W0730 16:57:11.087877 23980 master.cpp:2103] Ignoring unknown exited
executor 3 on slave 20140730-154530-1526966464-5050-22832-0 (CentOS-20)
I0730 16:57:11.088619 23985 master.cpp:2030] Status update TASK_LOST (UUID:
a3cccdd4-1d83-427f-bfc1-9dc42ec27b33) for task 3 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.92:5051
I0730 16:57:11.088718 23985 master.hpp:429] Removing task 3 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-0
(CentOS-20)
I0730 16:57:11.088917 23985 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: cpus(*):4; mem(*):1000;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-0 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.089830 23982 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.089877 23982 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.090384 23982 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.095751 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-15 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.095979 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-16 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:11.096027 23980 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:11.096071 23980 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:11.096146 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-17 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:11.096199 23980 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:11.096202 23978 master.hpp:411] Adding task 4 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1
(CentOS-21)
I0730 16:57:11.096222 23980 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
I0730 16:57:11.096242 23978 master.cpp:2451] Launching task 4 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1 (CentOS-21)
W0730 16:57:11.096488 23984 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:11.096544 23984 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:12.747547 23978 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:13.047899 23985 http.cpp:391] HTTP request for
'/master/state.json'
W0730 16:57:16.127420 23984 master.cpp:2103] Ignoring unknown exited
executor 4 on slave 20140730-154530-1526966464-5050-22832-1 (CentOS-21)
I0730 16:57:16.128063 23981 master.cpp:2030] Status update TASK_LOST (UUID:
ebb1fec7-2a30-4798-b6da-13f12422591d) for task 4 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.93:5051
I0730 16:57:16.128134 23981 master.hpp:429] Removing task 4 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1
(CentOS-21)
I0730 16:57:16.128263 23981 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: mem(*):1000; cpus(*):4;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-1 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.129209 23981 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.129315 23981 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.129767 23981 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.133862 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-18 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.134055 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-19 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:16.134093 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:16.134119 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:16.134213 23982 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-20 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:16.134223 23983 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:16.134240 23983 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
I0730 16:57:16.134268 23982 master.hpp:411] Adding task 5 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1
(CentOS-21)
I0730 16:57:16.134307 23982 master.cpp:2451] Launching task 5 of framework
20140730-165621-1526966464-5050-23977-0000 with resources cpus(*):2;
mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1 (CentOS-21)
W0730 16:57:16.134565 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:16.134588 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:19.047976 23978 http.cpp:391] HTTP request for
'/master/state.json'
W0730 16:57:21.144418 23984 master.cpp:2103] Ignoring unknown exited
executor 5 on slave 20140730-154530-1526966464-5050-22832-1 (CentOS-21)
I0730 16:57:21.145104 23981 master.cpp:2030] Status update TASK_LOST (UUID:
84107fc4-d997-4e9c-a256-00d30e5eb4f4) for task 5 of framework
20140730-165621-1526966464-5050-23977-0000 from slave(1)@192.168.3.93:5051
I0730 16:57:21.145171 23981 master.hpp:429] Removing task 5 with resources
cpus(*):2; mem(*):100 on slave 20140730-154530-1526966464-5050-22832-1
(CentOS-21)
I0730 16:57:21.145301 23981 hierarchical_allocator_process.hpp:637]
Recovered cpus(*):2; mem(*):100 (total allocatable: mem(*):1000; cpus(*):4;
disk(*):1.00684e+06; ports(*):[31000-32000]) on slave
20140730-154530-1526966464-5050-22832-1 from framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.146317 23981 master.cpp:1673] Reviving offers for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.146419 23985 hierarchical_allocator_process.hpp:661] Removed
filters for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.146939 23985 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.149528 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-21 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.149724 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-22 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:21.149725 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:21.149771 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:21.149870 23981 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-23 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:21.149893 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:21.149912 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:21.149981 23982 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:21.149996 23982 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:22.773200 23984 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:24.048961 23978 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:26.175562 23982 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:26.178474 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-24 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:26.178648 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-25 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:26.178674 23978 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:26.178706 23978 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:26.178809 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-26 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:26.178830 23978 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:26.178850 23978 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:26.178920 23978 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:26.178936 23978 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:30.046826 23981 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:32.180317 23978 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:32.183300 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-27 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:32.183472 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-28 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:32.183485 23984 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:32.183516 23984 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:32.183609 23985 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-29 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:32.183624 23984 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:32.183640 23984 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:32.183707 23984 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:32.183722 23984 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:32.794637 23982 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:35.048312 23983 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:37.185760 23979 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:37.188722 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-30 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:37.188920 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-31 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:37.188942 23981 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:37.188976 23981 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:37.189059 23978 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-32 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:37.189079 23981 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:37.189095 23981 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:37.189162 23981 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:37.189177 23981 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
I0730 16:57:41.046386 23984 http.cpp:391] HTTP request for
'/master/state.json'
I0730 16:57:42.190924 23982 master.cpp:2282] Sending 3 offers to framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:42.193842 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-33 ] on slave
20140730-154530-1526966464-5050-22832-2 (CentOS-22) for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:42.194020 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-34 ] on slave
20140730-154530-1526966464-5050-22832-0 (CentOS-20) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:42.194051 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:42.194079 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-2 for 5secs
I0730 16:57:42.194159 23983 master.cpp:1578] Processing reply for offers: [
20140730-165621-1526966464-5050-23977-35 ] on slave
20140730-154530-1526966464-5050-22832-1 (CentOS-21) for framework
20140730-165621-1526966464-5050-23977-0000
W0730 16:57:42.194185 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:42.194200 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-0 for 5secs
W0730 16:57:42.194269 23985 hierarchical_allocator_process.hpp:581] Using
the default value of 'refuse_seconds' to create the refused resources filter
because the input value is negative
I0730 16:57:42.194283 23985 hierarchical_allocator_process.hpp:590]
Framework 20140730-165621-1526966464-5050-23977-0000 filtered slave
20140730-154530-1526966464-5050-22832-1 for 5secs
below is log of one slave.
I0730 16:56:30.504299 20041 main.cpp:112] Build: 2014-07-28 15:34:46 by root
I0730 16:56:30.504802 20041 main.cpp:114] Version: 0.18.1
I0730 16:56:30.504838 20041 containerizer.cpp:180] Using isolation:
posix/cpu,posix/mem
I0730 16:56:30.506728 20041 main.cpp:135] Starting Mesos slave
I0730 16:56:30.507246 20041 slave.cpp:112] Slave started on
1)@192.168.3.93:5051
I0730 16:56:30.507508 20041 slave.cpp:122] Slave resources: mem(*):1000;
cpus(*):4; disk(*):1.00684e+06; ports(*):[31000-32000]
I0730 16:56:30.507694 20041 slave.cpp:150] Slave hostname: CentOS-21
I0730 16:56:30.507730 20041 slave.cpp:151] Slave checkpoint: true
I0730 16:56:30.510119 20046 state.cpp:33] Recovering state from
'/tmp/mesos/meta'
I0730 16:56:30.511029 20046 status_update_manager.cpp:188] Recovering status
update manager
I0730 16:56:30.511322 20049 mesos_containerizer.cpp:181] Recovering
containerizer
I0730 16:56:30.511744 20048 slave.cpp:2670] Finished recovery
I0730 16:56:30.512138 20048 slave.cpp:2702] Garbage collecting old slave
20140729-163345-1526966464-5050-10913-1
I0730 16:56:30.512262 20045 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140729-163345-1526966464-5050-10913-1' for gc
6.9999940718163days in the future
I0730 16:56:30.512291 20048 slave.cpp:2702] Garbage collecting old slave
20140729-133100-1526966464-5050-5688-1
I0730 16:56:30.512373 20045 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/20140729-163345-1526966464-5050-10913-1' for gc
6.99999407105481days in the future
I0730 16:56:30.512428 20045 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140729-133100-1526966464-5050-5688-1' for gc
6.99999406993185days in the future
I0730 16:56:30.512490 20045 gc.cpp:56] Scheduling
'/tmp/mesos/meta/slaves/20140729-133100-1526966464-5050-5688-1' for gc
6.99999406931556days in the future
I0730 16:56:30.512629 20045 slave.cpp:397] New master detected at
master@192.168.3.91:5050
I0730 16:56:30.512898 20044 status_update_manager.cpp:162] New master
detected at master@192.168.3.91:5050
I0730 16:56:30.512974 20045 slave.cpp:422] Detecting new master
I0730 16:56:30.514149 20044 slave.cpp:490] Re-registered with master
master@192.168.3.91:5050
I0730 16:57:11.916435 20046 slave.cpp:625] Got assigned task 4 for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.917269 20046 slave.cpp:734] Launching task 4 for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.919847 20046 slave.cpp:844] Queuing task '4' for executor 4
of framework '20140730-165621-1526966464-5050-23977-0000
I0730 16:57:11.919872 20047 mesos_containerizer.cpp:407] Starting container
'b01c0c15-62e6-4e33-a9ee-293bc19488da' for executor '4' of framework
'20140730-165621-1526966464-5050-23977-0000'
I0730 16:57:11.920572 20047 mesos_containerizer.cpp:528] Fetching URIs for
container 'b01c0c15-62e6-4e33-a9ee-293bc19488da' using command '/usr/bin/env
MESOS_EXECUTOR_URIS="hdfs://CentOS-19:9000/doc/spark-1.0.0.tgz+0X"
MESOS_WORK_DIRECTORY=/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/4/runs/b01c0c15-62e6-4e33-a9ee-293bc19488da
MESOS_USER=root /usr/local/libexec/mesos/mesos-fetcher'
I0730 16:57:15.925954 20042 launcher.cpp:120] Forked child with pid '20098'
for container 'b01c0c15-62e6-4e33-a9ee-293bc19488da'
I0730 16:57:15.927288 20043 slave.cpp:2001] Monitoring executor '4' of
framework '20140730-165621-1526966464-5050-23977-0000' in container
'b01c0c15-62e6-4e33-a9ee-293bc19488da'
I0730 16:57:16.926985 20044 mesos_containerizer.cpp:901] Executor for
container 'b01c0c15-62e6-4e33-a9ee-293bc19488da' has exited
I0730 16:57:16.927158 20044 mesos_containerizer.cpp:818] Destroying
container 'b01c0c15-62e6-4e33-a9ee-293bc19488da'
I0730 16:57:16.941853 20042 slave.cpp:2059] Executor '4' of framework
20140730-165621-1526966464-5050-23977-0000 has exited with status 1
I0730 16:57:16.943660 20042 slave.cpp:1669] Handling status update TASK_LOST
(UUID: ebb1fec7-2a30-4798-b6da-13f12422591d) for task 4 of framework
20140730-165621-1526966464-5050-23977-0000 from @0.0.0.0:0
I0730 16:57:16.946171 20046 status_update_manager.cpp:315] Received status
update TASK_LOST (UUID: ebb1fec7-2a30-4798-b6da-13f12422591d) for task 4 of
framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.946773 20046 status_update_manager.cpp:368] Forwarding status
update TASK_LOST (UUID: ebb1fec7-2a30-4798-b6da-13f12422591d) for task 4 of
framework 20140730-165621-1526966464-5050-23977-0000 to
master@192.168.3.91:5050
I0730 16:57:16.949275 20046 status_update_manager.cpp:393] Received status
update acknowledgement (UUID: ebb1fec7-2a30-4798-b6da-13f12422591d) for task
4 of framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.949676 20046 slave.cpp:2198] Cleaning up executor '4' of
framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.949949 20045 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/4/runs/b01c0c15-62e6-4e33-a9ee-293bc19488da'
for gc 6.99998900613333days in the future
I0730 16:57:16.950006 20046 slave.cpp:2273] Cleaning up framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.950121 20045 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/4'
for gc 6.99998900494222days in the future
I0730 16:57:16.950271 20047 status_update_manager.cpp:277] Closing status
update streams for framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.950453 20045 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000'
for gc 6.99998900153778days in the future
I0730 16:57:16.954074 20048 slave.cpp:625] Got assigned task 5 for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.954826 20048 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000'
from gc
I0730 16:57:16.955044 20048 slave.cpp:734] Launching task 5 for framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.957242 20046 mesos_containerizer.cpp:407] Starting container
'33dedf06-507b-4f0f-b59b-7890f876d3b4' for executor '5' of framework
'20140730-165621-1526966464-5050-23977-0000'
I0730 16:57:16.957444 20048 slave.cpp:844] Queuing task '5' for executor 5
of framework '20140730-165621-1526966464-5050-23977-0000
I0730 16:57:16.962373 20048 mesos_containerizer.cpp:528] Fetching URIs for
container '33dedf06-507b-4f0f-b59b-7890f876d3b4' using command '/usr/bin/env
MESOS_EXECUTOR_URIS="hdfs://CentOS-19:9000/doc/spark-1.0.0.tgz+0X"
MESOS_WORK_DIRECTORY=/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/5/runs/33dedf06-507b-4f0f-b59b-7890f876d3b4
MESOS_USER=root /usr/local/libexec/mesos/mesos-fetcher'
I0730 16:57:20.930588 20049 launcher.cpp:120] Forked child with pid '20145'
for container '33dedf06-507b-4f0f-b59b-7890f876d3b4'
I0730 16:57:20.932019 20043 slave.cpp:2001] Monitoring executor '5' of
framework '20140730-165621-1526966464-5050-23977-0000' in container
'33dedf06-507b-4f0f-b59b-7890f876d3b4'
I0730 16:57:21.931267 20043 mesos_containerizer.cpp:901] Executor for
container '33dedf06-507b-4f0f-b59b-7890f876d3b4' has exited
I0730 16:57:21.935158 20043 mesos_containerizer.cpp:818] Destroying
container '33dedf06-507b-4f0f-b59b-7890f876d3b4'
I0730 16:57:21.960533 20045 slave.cpp:2059] Executor '5' of framework
20140730-165621-1526966464-5050-23977-0000 has exited with status 1
I0730 16:57:21.962280 20045 slave.cpp:1669] Handling status update TASK_LOST
(UUID: 84107fc4-d997-4e9c-a256-00d30e5eb4f4) for task 5 of framework
20140730-165621-1526966464-5050-23977-0000 from @0.0.0.0:0
I0730 16:57:21.963222 20042 status_update_manager.cpp:315] Received status
update TASK_LOST (UUID: 84107fc4-d997-4e9c-a256-00d30e5eb4f4) for task 5 of
framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.963773 20042 status_update_manager.cpp:368] Forwarding status
update TASK_LOST (UUID: 84107fc4-d997-4e9c-a256-00d30e5eb4f4) for task 5 of
framework 20140730-165621-1526966464-5050-23977-0000 to
master@192.168.3.91:5050
I0730 16:57:21.966195 20042 status_update_manager.cpp:393] Received status
update acknowledgement (UUID: 84107fc4-d997-4e9c-a256-00d30e5eb4f4) for task
5 of framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.966434 20042 slave.cpp:2198] Cleaning up executor '5' of
framework 20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.966717 20049 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/5/runs/33dedf06-507b-4f0f-b59b-7890f876d3b4'
for gc 6.99998881231704days in the future
I0730 16:57:21.966872 20042 slave.cpp:2273] Cleaning up framework
20140730-165621-1526966464-5050-23977-0000
I0730 16:57:21.967042 20049 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000/executors/5'
for gc 6.99998880958518days in the future
I0730 16:57:21.967258 20049 gc.cpp:56] Scheduling
'/tmp/mesos/slaves/20140730-154530-1526966464-5050-22832-1/frameworks/20140730-165621-1526966464-5050-23977-0000'
for gc 6.99998880614519days in the future
I0730 16:57:21.967341 20042 status_update_manager.cpp:277] Closing status
update streams for framework 20140730-165621-1526966464-5050-23977-0000
spark running console print:
14/07/30 16:56:48 INFO server.AbstractConnector: Started
SelectChannelConnector@0.0.0.0:4041
14/07/30 16:56:48 INFO ui.SparkUI: Started SparkUI at http://CentOS-19:4041
14/07/30 16:56:48 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/07/30 16:56:49 INFO scheduler.EventLoggingListener: Logging events to
/tmp/spark-events/my_test.py-1406710609033
14/07/30 16:56:49 INFO util.Utils: Copying
/home/daijia/deal_three_word/my_test.py to
/tmp/spark-c8e9af2f-32b5-4bf0-9f57-c46dc82a4450/my_test.py
14/07/30 16:56:49 INFO spark.SparkContext: Added file
file:/home/daijia/deal_three_word/my_test.py at
http://192.168.3.91:42379/files/my_test.py with timestamp 1406710609772
I0730 16:56:49.882772 24123 sched.cpp:121] Version: 0.18.1
I0730 16:56:49.884660 24131 sched.cpp:217] New master detected at
master@192.168.3.91:5050
I0730 16:56:49.884770 24131 sched.cpp:225] No credentials provided.
Attempting to register without authentication
I0730 16:56:49.885520 24131 sched.cpp:391] Framework registered with
20140730-165621-1526966464-5050-23977-0000
14/07/30 16:56:49 INFO mesos.CoarseMesosSchedulerBackend: Registered as
framework ID 20140730-165621-1526966464-5050-23977-0000
14/07/30 16:56:50 INFO spark.SparkContext: Starting job: count at
/home/daijia/deal_three_word/my_test.py:27
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Got job 0 (count at
/home/daijia/deal_three_word/my_test.py:27) with 2 output partitions
(allowLocal=false)
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Final stage: Stage 0(count at
/home/daijia/deal_three_word/my_test.py:27)
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Parents of final stage:
List()
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Missing parents: List()
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Submitting Stage 0
(PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
14/07/30 16:56:50 INFO scheduler.DAGScheduler: Submitting 2 missing tasks
from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
14/07/30 16:56:50 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with
2 tasks
14/07/30 16:56:55 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 0 is
now TASK_LOST
14/07/30 16:57:00 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 1 is
now TASK_LOST
14/07/30 16:57:00 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140730-154530-1526966464-5050-22832-2"
due to too many failures; is Spark installed on it?
14/07/30 16:57:05 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 2 is
now TASK_LOST
14/07/30 16:57:05 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/07/30 16:57:11 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 3 is
now TASK_LOST
14/07/30 16:57:11 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140730-154530-1526966464-5050-22832-0"
due to too many failures; is Spark installed on it?
14/07/30 16:57:16 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 4 is
now TASK_LOST
14/07/30 16:57:20 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
14/07/30 16:57:21 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 5 is
now TASK_LOST
14/07/30 16:57:21 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
slave value: "20140730-154530-1526966464-5050-22832-1"
due to too many failures; is Spark installed on it?
14/07/30 16:57:35 WARN scheduler.TaskSchedulerImpl: Initial job has not
accepted any resources; check your cluster UI to ensure that workers are
registered and have sufficient memory
And advice will help.
Thanks,
Daijia
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-submit-Pyspark-job-in-mesos-tp10905p10950.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to submit Pyspark job in mesos?
Posted by Davies Liu <da...@databricks.com>.
Maybe mesos or spark was not configured correctly, could you check the log
files in mesos slaves?
It should log the reason when mesos can not lunch the executor.
On Tue, Jul 29, 2014 at 10:39 PM, daijia <ji...@intsig.com> wrote:
>
> Actually, it runs okay in my slaves deployed by standalone mode.
> When I switch to mesos, the error just occurs.
>
> Anyway, thanks for your reply and any ideas will help.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-submit-Pyspark-job-in-mesos-tp10905p10918.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to submit Pyspark job in mesos?
Posted by daijia <ji...@intsig.com>.
Actually, it runs okay in my slaves deployed by standalone mode.
When I switch to mesos, the error just occurs.
Anyway, thanks for your reply and any ideas will help.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-submit-Pyspark-job-in-mesos-tp10905p10918.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How to submit Pyspark job in mesos?
Posted by Davies Liu <da...@databricks.com>.
On Tue, Jul 29, 2014 at 6:42 PM, daijia <ji...@intsig.com> wrote:
> Dear all,
>
> I have spark1.0.0 and mesos0.18.1. After setting in mesos and spark
> and starting the mesos cluster, I try to run the pyspark job by the command
> below:
>
> spark-submit /path/to/my_pyspark_job.py --master
> mesos://192.168.0.21:5050
>
> It occurs error below:
>
> 14/07/29 18:40:49 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/29 18:40:49 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4041
> 14/07/29 18:40:49 INFO ui.SparkUI: Started SparkUI at http://CentOS-19:4041
> 14/07/29 18:40:49 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/07/29 18:40:50 INFO scheduler.EventLoggingListener: Logging events to
> /tmp/spark-events/my_test.py-1406630449771
> 14/07/29 18:40:50 INFO util.Utils: Copying
> /home/daijia/deal_three_word/my_test.py to
> /tmp/spark-4365b01d-b57a-4abb-b39c-cb57b83a28ce/my_test.py
> 14/07/29 18:40:50 INFO spark.SparkContext: Added file
> file:/home/daijia/deal_three_word/my_test.py at
> http://192.168.3.91:51188/files/my_test.py with timestamp 1406630450333
> I0729 18:40:50.440551 15033 sched.cpp:121] Version: 0.18.1
> I0729 18:40:50.442450 15035 sched.cpp:217] New master detected at
> master@192.168.3.91:5050
> I0729 18:40:50.442570 15035 sched.cpp:225] No credentials provided.
> Attempting to register without authentication
> I0729 18:40:50.443234 15036 sched.cpp:391] Framework registered with
> 20140729-174911-1526966464-5050-13758-0006
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Registered as
> framework ID 20140729-174911-1526966464-5050-13758-0006
> 14/07/29 18:40:50 INFO spark.SparkContext: Starting job: count at
> /home/daijia/deal_three_word/my_test.py:27
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 0 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 1 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 3 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
> slave value: "20140729-163345-1526966464-5050-10913-0"
> due to too many failures; is Spark installed on it?
The Spark Executor can not start on mesos slaves, you can check the
logs on mesos slaves.
Maybe you forgot to install Spark on slaves?
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 2 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
> slave value: "20140729-163345-1526966464-5050-10913-2"
> due to too many failures; is Spark installed on it?
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Got job 0 (count at
> /home/daijia/deal_three_word/my_test.py:27) with 2 output partitions
> (allowLocal=false)
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Final stage: Stage 0(count at
> /home/daijia/deal_three_word/my_test.py:27)
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Parents of final stage:
> List()
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Missing parents: List()
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 4 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Submitting Stage 0
> (PythonRDD[1] at RDD at PythonRDD.scala:37), which has no missing parents
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Mesos task 5 is
> now TASK_LOST
> 14/07/29 18:40:50 INFO mesos.CoarseMesosSchedulerBackend: Blacklisting Mesos
> slave value: "20140729-163345-1526966464-5050-10913-1"
> due to too many failures; is Spark installed on it?
> 14/07/29 18:40:50 INFO scheduler.DAGScheduler: Submitting 2 missing tasks
> from Stage 0 (PythonRDD[1] at RDD at PythonRDD.scala:37)
> 14/07/29 18:40:50 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with
> 2 tasks
> 14/07/29 18:41:05 WARN scheduler.TaskSchedulerImpl: Initial job has not
> accepted any resources; check your cluster UI to ensure that workers are
> registered and have sufficient memory
> 14/07/29 18:41:20 WARN scheduler.TaskSchedulerImpl: Initial job has not
> accepted any resources; check your cluster UI to ensure that workers are
> registered and have sufficient memory
> 14/07/29 18:41:20 WARN scheduler.TaskSchedulerImpl: Initial job has not
> accepted any resources; check your cluster UI to ensure that workers are
> registered and have sufficient memory
>
> It just repeats the last message.
> Here is my python scirpt:
>
> #!/usr/bin/env python
> #coding=utf-8
> from pyspark import SparkContext
> sc = SparkContext()
> temp = []
> for index in range(1000):
> temp.append(index)
> sc.parallelize(temp).count()
>
>
> So, the running command is right? Or some other reasons lead to the
> problem.
>
> Thanks in advance,
> Daijia
>
>
>
>
>
>
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-submit-Pyspark-job-in-mesos-tp10905.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.