You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by Yingbo Wang <yb...@gmail.com> on 2013/12/20 08:20:24 UTC
Instruction on setting up dev environment on my Mac machine
hi,
Hope you guys have a fun night at city hall.
While you might be drunk, can anyone help me a little bit on setting up
the local dev environment to debug aurora ?
My current set up.
Machine A: 64bit ubuntu server running mesos-master,
Machine B: 64bit ubuntu server running mesos-slave, thermo
observer server;
Machine C: 32bit ubuntu server running zookeeper.
Machine D: Mac air for developing aurora.
I tried to run aurora-scheduler on Machine A and also created a hello
job from machine A. the job was submitted successfully. however the job
keeps staying at "PENDING" status. i checked thermo observer from port
1336, it says "No matching hosts".
I spent a couple of hours and still can't figure out the reason so
I decided to settup the dev environment on my Mac so that I can trace the
error in IntelliJ. Can someone give me some hint on that? I am not quite
familiar with IntelliJ and java.
best
Yingbo
Re: Instruction on setting up dev environment on my Mac machine
Posted by Zameer Manji <zm...@gmail.com>.
I am confident that the job cannot be scheduled because aurora does not
consider offers from slaves where the attributes 'rack' and 'host' are not
set. Can you please ensure your slave has the attributes 'host' and 'rack'
set?
You can launch the slave like so:
https://github.com/apache/incubator-aurora/blob/master/examples/vagrant/provision-mesos-slave.sh#L12
On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman <wi...@twopensource.com>wrote:
> There is an executor_id in the state there, so it looks like it is managing
> to launch the tasks. Since you've listed your job as a Service(), it means
> that every task will be rescheduled after it exits, and since it looks like
> your task just immediately returns, it's probably launching dozens of them
> per minute if not more. You can change this behavior by changing Service
> to Job. (Service is just an alias for Job(service=True).)
>
> ~brian
>
>
>
> On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
>
> > Thanks Maxim.
> >
> > did you mean hello.aurora?
> >
> >
> > hello = Process(
> > name = 'hello',
> > cmdline = """
> > echo "ls file " ;
> > whoami;
> > hostname;
> > """)
> >
> > task = SequentialTask(
> > processes = [hello],
> > resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
> >
> > jobs = [Service(
> > task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
> > name = 'hello')]
> >
> > The slave machine has enough resource to satisfy this request.
> >
> > I did encounter an issue saying "Constraint Not Satisfied, host". After I
> > change my hostname of both master and slave machines, this problem seems
> to
> > be solved, and now the problem becomes "No matching hosts"
> >
> >
> >
> > Here's the offers that scheduler received:
> >
> > [
> >
> > -
> > {
> > - id: "201312200632-302033088-5050-15916-396",
> > - framework_id: "201312110654-302033088-5050-31688-0000",
> > - slave_id: "201312170824-302033088-5050-25318-0",
> > - hostname: "192.168.0.11",
> > - resources:
> > [
> > -
> > {
> > - name: "cpus",
> > - scalar: 7.75
> > },
> > -
> > {
> > - name: "disk",
> > - scalar: 917136
> > },
> > -
> > {
> > - name: "mem",
> > - scalar: 14814
> > },
> > -
> > {
> > - name: "ports",
> > - ranges:
> > [
> > - "31000-32000"
> > ]
> > }
> > ],
> > - attributes:
> > [
> > -
> > {
> > - name: "host",
> > - text: "192.168.0.11"
> > },
> > -
> > {
> > - name: "rack",
> > - text: "a"
> > }
> > ],
> > - executor_ids:
> > [
> > -
> >
> >
> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
> > ]
> > }
> >
> > ]
> >
> >
> > and the /vars are:
> >
> >
> > async_tasks_completed 393
> > both_instance_ids_set 0
> > build_date null
> > build_git_branchname null
> > build_git_revision null
> > build_git_revision_number null
> > build_git_tag null
> > build_machine null
> > build_path null
> > build_time null
> > build_timestamp null
> > build_user_name null
> > cron_job_launch_failures 0
> > cron_jobs_triggered 0
> > cron_num_pending_runs 0
> > empty_slots_large 0
> > empty_slots_medium 1
> > empty_slots_small 7
> > empty_slots_xlarge 0
> > framework_registered 1
> > http_200_responses_events 22
> > http_200_responses_events_per_sec 0.0
> > http_200_responses_nanos_per_event 0.0
> > http_200_responses_nanos_total 981542679
> > http_200_responses_nanos_total_per_sec 0.0
> > instance_ids_inconsistent 0
> > jvm_available_processors 4
> > jvm_class_loaded_count 6658
> > jvm_class_total_loaded_count 6658
> > jvm_class_unloaded_count 0
> > jvm_gc_PS_MarkSweep_collection_count 0
> > jvm_gc_PS_MarkSweep_collection_time_ms 0
> > jvm_gc_PS_Scavenge_collection_count 7
> > jvm_gc_PS_Scavenge_collection_time_ms 534
> > jvm_gc_collection_count 7
> > jvm_gc_collection_time_ms 534
> > jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
> > jvm_memory_free_mb 688
> > jvm_memory_heap_mb_committed 968
> > jvm_memory_heap_mb_max 968
> > jvm_memory_heap_mb_used 279
> > jvm_memory_max_mb 968
> > jvm_memory_mb_total 968
> > jvm_memory_non_heap_mb_committed 39
> > jvm_memory_non_heap_mb_max 130
> > jvm_memory_non_heap_mb_used 38
> > jvm_prop_awt_toolkit sun.awt.X11.XToolkit
> > jvm_prop_file_encoding UTF-8
> > jvm_prop_file_encoding_pkg sun.io
> > jvm_prop_file_separator /
> > jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
> > jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
> > jvm_prop_java_class_path
> >
> >
> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
> > jvm_prop_java_class_version 51.0
> > jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
> > jvm_prop_java_ext_dirs
> > /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
> > jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
> > jvm_prop_java_io_tmpdir /tmp
> > jvm_prop_java_library_path /usr/local/lib
> > jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
> > jvm_prop_java_runtime_version 1.7.0_45-b18
> > jvm_prop_java_specification_name Java Platform API Specification
> > jvm_prop_java_specification_vendor Oracle Corporation
> > jvm_prop_java_specification_version 1.7
> > jvm_prop_java_vendor Oracle Corporation
> > jvm_prop_java_vendor_url http://java.oracle.com/
> > jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
> > jvm_prop_java_version 1.7.0_45
> > jvm_prop_java_vm_info mixed mode
> > jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
> > jvm_prop_java_vm_specification_name Java Virtual Machine Specification
> > jvm_prop_java_vm_specification_vendor Oracle Corporation
> > jvm_prop_java_vm_specification_version 1.7
> > jvm_prop_java_vm_vendor Oracle Corporation
> > jvm_prop_java_vm_version 24.45-b08
> > jvm_prop_line_separator
> >
> > jvm_prop_os_arch amd64
> > jvm_prop_os_name Linux
> > jvm_prop_os_version 3.5.0-44-generic
> > jvm_prop_path_separator :
> > jvm_prop_sun_arch_data_model 64
> > jvm_prop_sun_boot_class_path
> >
> >
> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
> > jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
> > jvm_prop_sun_cpu_endian little
> > jvm_prop_sun_cpu_isalist
> > jvm_prop_sun_io_unicode_encoding UnicodeLittle
> > jvm_prop_sun_java_command
> > com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
> > -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
> > -zk_endpoints=192.168.0.13:2181
> > -mesos_master_address=zk://192.168.0.13:2181/mesos/master
> > -serverset_path=/aurora/scheduler
> > -native_log_zk_group_path=/aurora/replicated-log
> > -native_log_file_path=/usr/local/aurora-scheduler/db
> > -backup_dir=/usr/local/aurora-scheduler/backups
> > -thermos_executor_path=/usr/local/bin/thermos_executor
> > -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
> > jvm_prop_sun_java_launcher SUN_STANDARD
> > jvm_prop_sun_jnu_encoding UTF-8
> > jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
> > jvm_prop_sun_os_patch_level unknown
> > jvm_prop_user_country US
> > jvm_prop_user_dir /home/lychee
> > jvm_prop_user_home /root
> > jvm_prop_user_language en
> > jvm_prop_user_name root
> > jvm_prop_user_timezone Etc/UTC
> > jvm_threads_active 21
> > jvm_threads_daemon 12
> > jvm_threads_peak 25
> > jvm_threads_started 867
> > jvm_time_ms 1387559947214
> > jvm_uptime_secs 38781
> > log_compressed_entry_bytes_saved 2537635
> > logging_dir /root
> > mem_storage_consistent_read_operation_events 619
> > mem_storage_consistent_read_operation_events_per_sec 0.0
> > mem_storage_consistent_read_operation_nanos_per_event 0.0
> > mem_storage_consistent_read_operation_nanos_total 272565131
> > mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
> > mem_storage_delete_all_tasks_events 1
> > mem_storage_delete_all_tasks_events_per_sec 0.0
> > mem_storage_delete_all_tasks_nanos_per_event 0.0
> > mem_storage_delete_all_tasks_nanos_total 2323293
> > mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
> > mem_storage_delete_tasks_events 179
> > mem_storage_delete_tasks_events_per_sec 0.0
> > mem_storage_delete_tasks_nanos_per_event 0.0
> > mem_storage_delete_tasks_nanos_total 8638495
> > mem_storage_delete_tasks_nanos_total_per_sec 0.0
> > mem_storage_fetch_tasks_events 3219
> > mem_storage_fetch_tasks_events_per_sec 0.0
> > mem_storage_fetch_tasks_nanos_per_event 0.0
> > mem_storage_fetch_tasks_nanos_total 193613049
> > mem_storage_fetch_tasks_nanos_total_per_sec 0.0
> > mem_storage_mutate_tasks_events 677
> > mem_storage_mutate_tasks_events_per_sec 0.0
> > mem_storage_mutate_tasks_nanos_per_event 0.0
> > mem_storage_mutate_tasks_nanos_total 149943165
> > mem_storage_mutate_tasks_nanos_total_per_sec 0.0
> > mem_storage_save_tasks_events 306
> > mem_storage_save_tasks_events_per_sec 0.0
> > mem_storage_save_tasks_nanos_per_event 0.0
> > mem_storage_save_tasks_nanos_total 165988968
> > mem_storage_save_tasks_nanos_total_per_sec 0.0
> > mem_storage_weakly_consistent_read_operation_events 1538
> > mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
> > mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
> > mem_storage_weakly_consistent_read_operation_nanos_total 138584844
> > mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
> > mem_storage_write_operation_events 2932
> > mem_storage_write_operation_events_per_sec 0.0
> > mem_storage_write_operation_nanos_per_event 0.0
> > mem_storage_write_operation_nanos_total 28077154879
> > mem_storage_write_operation_nanos_total_per_sec 0.0
> > new_instance_id_set 0
> > offer_accept_races 0
> > old_instance_id_set 0
> > outstanding_offers 1
> > preemptor_attempts 0
> > preemptor_failed_preemptions 0
> > preemptor_no_slots_found 0
> > preemptor_tasks_preempted 0
> > process_cpu_cores_utilized 0.0
> > process_cpu_time_nanos 73230000000
> > process_max_fd_count 4096
> > process_open_fd_count 157
> > read_lock_wait_nanos 545041
> > resources_allocated_quota_cpu 0
> > resources_allocated_quota_disk_gb 0
> > resources_allocated_quota_ram_gb 0
> > resources_dedicated_consumed_cpu 0
> > resources_dedicated_consumed_disk_gb 0
> > resources_dedicated_consumed_ram_gb 0
> > resources_free_pool_consumed_cpu 1
> > resources_free_pool_consumed_disk_gb 0
> > resources_free_pool_consumed_ram_gb 0
> > resources_quota_consumed_cpu 0
> > resources_quota_consumed_disk_gb 0
> > resources_quota_consumed_ram_gb 0
> > resources_total_consumed_cpu 1
> > resources_total_consumed_disk_gb 0
> > resources_total_consumed_ram_gb 0
> > schedule_attempts_failed 0
> > schedule_attempts_fired 135
> > schedule_queue_size 1
> > scheduler_backup_failed 0
> > scheduler_backup_success 10
> > scheduler_driver_kill_failures 0
> > scheduler_failed_offers 0
> > scheduler_framework_disconnects 0
> > scheduler_framework_reregisters 0
> > scheduler_illegal_task_state_transitions 200
> > scheduler_log_bad_frames_read 0
> > scheduler_log_bytes_read 292813
> > scheduler_log_bytes_written 1080061
> > scheduler_log_deflated_entries_read 1
> > scheduler_log_entries_read 175
> > scheduler_log_entries_written 555
> > scheduler_log_native_append_events 556
> > scheduler_log_native_append_events_per_sec 0.0
> > scheduler_log_native_append_failures 0
> > scheduler_log_native_append_nanos_per_event 0.0
> > scheduler_log_native_append_nanos_total 25082628348
> > scheduler_log_native_append_nanos_total_per_sec 0.0
> > scheduler_log_native_append_timeouts 0
> > scheduler_log_native_native_entries_skipped 1
> > scheduler_log_native_read_events 176
> > scheduler_log_native_read_events_per_sec 0.0
> > scheduler_log_native_read_failures 0
> > scheduler_log_native_read_nanos_per_event 0.0
> > scheduler_log_native_read_nanos_total 73255842
> > scheduler_log_native_read_nanos_total_per_sec 0.0
> > scheduler_log_native_read_timeouts 0
> > scheduler_log_native_truncate_events 10
> > scheduler_log_native_truncate_events_per_sec 0.0
> > scheduler_log_native_truncate_failures 0
> > scheduler_log_native_truncate_nanos_per_event 0.0
> > scheduler_log_native_truncate_nanos_total 276600750
> > scheduler_log_native_truncate_nanos_total_per_sec 0.0
> > scheduler_log_native_truncate_timeouts 0
> > scheduler_log_recover_events 1
> > scheduler_log_recover_events_per_sec 0.0
> > scheduler_log_recover_nanos_per_event 0.0
> > scheduler_log_recover_nanos_total 565333612
> > scheduler_log_recover_nanos_total_per_sec 0.0
> > scheduler_log_save_framework_id_events 2
> > scheduler_log_save_framework_id_events_per_sec 0.0
> > scheduler_log_save_framework_id_nanos_per_event 0.0
> > scheduler_log_save_framework_id_nanos_total 437292
> > scheduler_log_save_framework_id_nanos_total_per_sec 0.0
> > scheduler_log_snapshot_events 10
> > scheduler_log_snapshot_events_per_sec 0.0
> > scheduler_log_snapshot_nanos_per_event 0.0
> > scheduler_log_snapshot_nanos_total 961465418
> > scheduler_log_snapshot_nanos_total_per_sec 0.0
> > scheduler_log_snapshot_persist_events 10
> > scheduler_log_snapshot_persist_events_per_sec 0.0
> > scheduler_log_snapshot_persist_nanos_per_event 0.0
> > scheduler_log_snapshot_persist_nanos_total 840444765
> > scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
> > scheduler_log_snapshots 10
> > scheduler_log_tasks_mutate_events 677
> > scheduler_log_tasks_mutate_events_per_sec 0.0
> > scheduler_log_tasks_mutate_nanos_per_event 0.0
> > scheduler_log_tasks_mutate_nanos_total 182346727
> > scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
> > scheduler_log_tasks_remove_events 179
> > scheduler_log_tasks_remove_events_per_sec 0.0
> > scheduler_log_tasks_remove_nanos_per_event 0.0
> > scheduler_log_tasks_remove_nanos_total 13588163
> > scheduler_log_tasks_remove_nanos_total_per_sec 0.0
> > scheduler_log_tasks_save_events 305
> > scheduler_log_tasks_save_events_per_sec 0.0
> > scheduler_log_tasks_save_nanos_per_event 0.0
> > scheduler_log_tasks_save_nanos_total 114479852
> > scheduler_log_tasks_save_nanos_total_per_sec 0.0
> > scheduler_log_un_snapshotted_transactions 36
> > scheduler_lost_executors 0
> > scheduler_max_ASSIGNED_waiting_ms 0
> > scheduler_max_KILLING_waiting_ms 0
> > scheduler_max_PREEMPTING_waiting_ms 0
> > scheduler_max_RESTARTING_waiting_ms 0
> > scheduler_resource_offers 397
> > scheduler_resource_offers_events 397
> > scheduler_resource_offers_events_per_sec 0.0
> > scheduler_resource_offers_nanos_per_event 0.0
> > scheduler_resource_offers_nanos_total 305282357
> > scheduler_resource_offers_nanos_total_per_sec 0.0
> > scheduler_save_host_attribute_events 397
> > scheduler_save_host_attribute_events_per_sec 0.0
> > scheduler_save_host_attribute_nanos_per_event 0.0
> > scheduler_save_host_attribute_nanos_total 12466197
> > scheduler_save_host_attribute_nanos_total_per_sec 0.0
> > scheduler_status_update_events 426
> > scheduler_status_update_events_per_sec 0.0
> > scheduler_status_update_nanos_per_event 0.0
> > scheduler_status_update_nanos_total 19334004116
> > scheduler_status_update_nanos_total_per_sec 0.0
> > scheduler_status_updates 0
> > scheduler_thrift_createJob_events 2
> > scheduler_thrift_createJob_events_per_sec 0.0
> > scheduler_thrift_createJob_nanos_per_event 0.0
> > scheduler_thrift_createJob_nanos_total 54102009
> > scheduler_thrift_createJob_nanos_total_per_sec 0.0
> > scheduler_thrift_getTasksStatus_events 2
> > scheduler_thrift_getTasksStatus_events_per_sec 0.0
> > scheduler_thrift_getTasksStatus_nanos_per_event 0.0
> > scheduler_thrift_getTasksStatus_nanos_total 5684762
> > scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
> > scheduler_thrift_getVersion_events 4
> > scheduler_thrift_getVersion_events_per_sec 0.0
> > scheduler_thrift_getVersion_nanos_per_event 0.0
> > scheduler_thrift_getVersion_nanos_total 13854714
> > scheduler_thrift_getVersion_nanos_total_per_sec 0.0
> > scheduler_thrift_killTasks_events 2
> > scheduler_thrift_killTasks_events_per_sec 0.0
> > scheduler_thrift_killTasks_nanos_per_event 0.0
> > scheduler_thrift_killTasks_nanos_total 186407312
> > scheduler_thrift_killTasks_nanos_total_per_sec 0.0
> > shard_sanity_check_failures 0
> > snapshot_apply_events 1
> > snapshot_apply_events_per_sec 0.0
> > snapshot_apply_nanos_per_event 0.0
> > snapshot_apply_nanos_total 93051831
> > snapshot_apply_nanos_total_per_sec 0.0
> > snapshot_create_events 10
> > snapshot_create_events_per_sec 0.0
> > snapshot_create_nanos_per_event 0.0
> > snapshot_create_nanos_total 37066472
> > snapshot_create_nanos_total_per_sec 0.0
> > system_env_GLOG_v 0
> > system_env_HOME /home/lychee
> > system_env_JAVA_OPTS -server -Xmx1g -Xms1g
> > -Djava.library.path=/usr/local/lib
> > system_env_LANG en_US.UTF-8
> > system_env_LIBPROCESS_IP 192.168.0.18
> > system_env_LIBPROCESS_PORT 8083
> > system_env_LOGNAME root
> > system_env_MAIL /var/mail/root
> > system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> > system_env_OLDPWD /usr/local/aurora-scheduler
> > system_env_PATH
> > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > system_env_PWD /home/lychee
> > system_env_SHELL /bin/bash
> > system_env_SHLVL 0
> > system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
> > system_env_SUDO_GID 1002
> > system_env_SUDO_UID 1002
> > system_env_SUDO_USER lychee
> > system_env_TERM xterm
> > system_env_USER root
> > system_env_USERNAME root
> > system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
> > system_free_physical_memory_mb 148
> > system_free_swap_mb 8063
> > system_load_avg 2.22
> > task_queries_all 45
> > task_queries_by_id 3668
> > task_queries_by_job 183
> > task_schedule_attempt_events 135
> > task_schedule_attempt_events_per_sec 0.0
> > task_schedule_attempt_nanos_per_event 0.0
> > task_schedule_attempt_nanos_total 6289005821
> > task_schedule_attempt_nanos_total_per_sec 0.0
> > task_store_ASSIGNED 0
> > task_store_FAILED 0
> > task_store_FINISHED 200
> > task_store_INIT 0
> > task_store_KILLED 0
> > task_store_KILLING 0
> > task_store_LOST 0
> > task_store_PENDING 1
> > task_store_PREEMPTING 0
> > task_store_RESTARTING 0
> > task_store_RUNNING 0
> > task_store_STARTING 0
> > task_store_THROTTLED 0
> > task_store_UNKNOWN 0
> > timed_out_tasks 0
> > timeout_queue_size 338
> > transient_states 0
> > uncaught_exceptions 0
> > variable_scrape_events 38778
> > variable_scrape_events_per_sec 1.0000103021061324
> > variable_scrape_micros_per_event 455.0001369564521
> > variable_scrape_micros_total 18809481
> > variable_scrape_micros_total_per_sec 455.00482441615327
> > work_queue_depth 0
> > write_lock_wait_nanos 13138744
> >
> >
> >
> >
> > On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
> > >wrote:
> >
> > > Hi Yingbo,
> > >
> > > Do you mind sharing your .aurora file? Perhaps your are requesting
> > > Resources or Constraints that are infeasible for your local setup to
> > > satisfy?
> > >
> > > Also, you might want to check that your scheduler is receiving offers
> > from
> > > mesos master. You could check your scheduler log file to see if any
> > offers
> > > are coming. Alternatively, you might want to look at
> > > "scheduler_resource_offers" stat exposed by the scheduler /vars
> endpoint
> > > (http://<your_local_scheduler_ip:port>/vars).
> > >
> > > Thanks,
> > > Maxim
> > >
> > >
> > >
> > > On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> > >
> > > > hi,
> > > > Hope you guys have a fun night at city hall.
> > > > While you might be drunk, can anyone help me a little bit on
> setting
> > > up
> > > > the local dev environment to debug aurora ?
> > > > My current set up.
> > > > Machine A: 64bit ubuntu server running mesos-master,
> > > > Machine B: 64bit ubuntu server running mesos-slave, thermo
> > > > observer server;
> > > > Machine C: 32bit ubuntu server running zookeeper.
> > > > Machine D: Mac air for developing aurora.
> > > >
> > > > I tried to run aurora-scheduler on Machine A and also created a
> > > hello
> > > > job from machine A. the job was submitted successfully. however the
> job
> > > > keeps staying at "PENDING" status. i checked thermo observer from
> port
> > > > 1336, it says "No matching hosts".
> > > >
> > > > I spent a couple of hours and still can't figure out the
> reason
> > so
> > > > I decided to settup the dev environment on my Mac so that I can trace
> > the
> > > > error in IntelliJ. Can someone give me some hint on that? I am not
> > quite
> > > > familiar with IntelliJ and java.
> > > >
> > > >
> > > >
> > > > best
> > > > Yingbo
> > >
> > >
> >
>
--
Zameer Manji
Re: Instruction on setting up dev environment on my Mac machine
Posted by Brian Wickman <wi...@twopensource.com>.
I think this is the default message as the task goes into PENDING, which
will stick until the scheduler has had an opportunity to drain the PENDING
queue. If it were really PENDING due to constraint reasons, it would at
least say something along the lines of "No matching hosts: rack" or
similar. Feel free to file a ticket to improve the messaging.
On Fri, Dec 20, 2013 at 12:01 PM, Yingbo Wang <yb...@gmail.com> wrote:
> It is a good catch! Thanks Brian.
>
> The task indeed restarts every 5 minutes. I guess the job just doesn't
> can't find the running service that's why it says "Not matching hosts"?
> I will try it to see if it works.
>
>
>
>
> On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman <wickman@twopensource.com
> >wrote:
>
> > There is an executor_id in the state there, so it looks like it is
> managing
> > to launch the tasks. Since you've listed your job as a Service(), it
> means
> > that every task will be rescheduled after it exits, and since it looks
> like
> > your task just immediately returns, it's probably launching dozens of
> them
> > per minute if not more. You can change this behavior by changing Service
> > to Job. (Service is just an alias for Job(service=True).)
> >
> > ~brian
> >
> >
> >
> > On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
> >
> > > Thanks Maxim.
> > >
> > > did you mean hello.aurora?
> > >
> > >
> > > hello = Process(
> > > name = 'hello',
> > > cmdline = """
> > > echo "ls file " ;
> > > whoami;
> > > hostname;
> > > """)
> > >
> > > task = SequentialTask(
> > > processes = [hello],
> > > resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
> > >
> > > jobs = [Service(
> > > task = task, cluster = 'lychee', role = 'lychee', environment =
> 'prod',
> > > name = 'hello')]
> > >
> > > The slave machine has enough resource to satisfy this request.
> > >
> > > I did encounter an issue saying "Constraint Not Satisfied, host".
> After I
> > > change my hostname of both master and slave machines, this problem
> seems
> > to
> > > be solved, and now the problem becomes "No matching hosts"
> > >
> > >
> > >
> > > Here's the offers that scheduler received:
> > >
> > > [
> > >
> > > -
> > > {
> > > - id: "201312200632-302033088-5050-15916-396",
> > > - framework_id: "201312110654-302033088-5050-31688-0000",
> > > - slave_id: "201312170824-302033088-5050-25318-0",
> > > - hostname: "192.168.0.11",
> > > - resources:
> > > [
> > > -
> > > {
> > > - name: "cpus",
> > > - scalar: 7.75
> > > },
> > > -
> > > {
> > > - name: "disk",
> > > - scalar: 917136
> > > },
> > > -
> > > {
> > > - name: "mem",
> > > - scalar: 14814
> > > },
> > > -
> > > {
> > > - name: "ports",
> > > - ranges:
> > > [
> > > - "31000-32000"
> > > ]
> > > }
> > > ],
> > > - attributes:
> > > [
> > > -
> > > {
> > > - name: "host",
> > > - text: "192.168.0.11"
> > > },
> > > -
> > > {
> > > - name: "rack",
> > > - text: "a"
> > > }
> > > ],
> > > - executor_ids:
> > > [
> > > -
> > >
> > >
> >
> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
> > > ]
> > > }
> > >
> > > ]
> > >
> > >
> > > and the /vars are:
> > >
> > >
> > > async_tasks_completed 393
> > > both_instance_ids_set 0
> > > build_date null
> > > build_git_branchname null
> > > build_git_revision null
> > > build_git_revision_number null
> > > build_git_tag null
> > > build_machine null
> > > build_path null
> > > build_time null
> > > build_timestamp null
> > > build_user_name null
> > > cron_job_launch_failures 0
> > > cron_jobs_triggered 0
> > > cron_num_pending_runs 0
> > > empty_slots_large 0
> > > empty_slots_medium 1
> > > empty_slots_small 7
> > > empty_slots_xlarge 0
> > > framework_registered 1
> > > http_200_responses_events 22
> > > http_200_responses_events_per_sec 0.0
> > > http_200_responses_nanos_per_event 0.0
> > > http_200_responses_nanos_total 981542679
> > > http_200_responses_nanos_total_per_sec 0.0
> > > instance_ids_inconsistent 0
> > > jvm_available_processors 4
> > > jvm_class_loaded_count 6658
> > > jvm_class_total_loaded_count 6658
> > > jvm_class_unloaded_count 0
> > > jvm_gc_PS_MarkSweep_collection_count 0
> > > jvm_gc_PS_MarkSweep_collection_time_ms 0
> > > jvm_gc_PS_Scavenge_collection_count 7
> > > jvm_gc_PS_Scavenge_collection_time_ms 534
> > > jvm_gc_collection_count 7
> > > jvm_gc_collection_time_ms 534
> > > jvm_input_arguments [-Xmx1g, -Xms1g,
> -Djava.library.path=/usr/local/lib]
> > > jvm_memory_free_mb 688
> > > jvm_memory_heap_mb_committed 968
> > > jvm_memory_heap_mb_max 968
> > > jvm_memory_heap_mb_used 279
> > > jvm_memory_max_mb 968
> > > jvm_memory_mb_total 968
> > > jvm_memory_non_heap_mb_committed 39
> > > jvm_memory_non_heap_mb_max 130
> > > jvm_memory_non_heap_mb_used 38
> > > jvm_prop_awt_toolkit sun.awt.X11.XToolkit
> > > jvm_prop_file_encoding UTF-8
> > > jvm_prop_file_encoding_pkg sun.io
> > > jvm_prop_file_separator /
> > > jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
> > > jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
> > > jvm_prop_java_class_path
> > >
> > >
> >
> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
> > > jvm_prop_java_class_version 51.0
> > > jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
> > > jvm_prop_java_ext_dirs
> > > /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
> > > jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
> > > jvm_prop_java_io_tmpdir /tmp
> > > jvm_prop_java_library_path /usr/local/lib
> > > jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
> > > jvm_prop_java_runtime_version 1.7.0_45-b18
> > > jvm_prop_java_specification_name Java Platform API Specification
> > > jvm_prop_java_specification_vendor Oracle Corporation
> > > jvm_prop_java_specification_version 1.7
> > > jvm_prop_java_vendor Oracle Corporation
> > > jvm_prop_java_vendor_url http://java.oracle.com/
> > > jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
> > > jvm_prop_java_version 1.7.0_45
> > > jvm_prop_java_vm_info mixed mode
> > > jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
> > > jvm_prop_java_vm_specification_name Java Virtual Machine Specification
> > > jvm_prop_java_vm_specification_vendor Oracle Corporation
> > > jvm_prop_java_vm_specification_version 1.7
> > > jvm_prop_java_vm_vendor Oracle Corporation
> > > jvm_prop_java_vm_version 24.45-b08
> > > jvm_prop_line_separator
> > >
> > > jvm_prop_os_arch amd64
> > > jvm_prop_os_name Linux
> > > jvm_prop_os_version 3.5.0-44-generic
> > > jvm_prop_path_separator :
> > > jvm_prop_sun_arch_data_model 64
> > > jvm_prop_sun_boot_class_path
> > >
> > >
> >
> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
> > > jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
> > > jvm_prop_sun_cpu_endian little
> > > jvm_prop_sun_cpu_isalist
> > > jvm_prop_sun_io_unicode_encoding UnicodeLittle
> > > jvm_prop_sun_java_command
> > > com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
> > > -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
> > > -zk_endpoints=192.168.0.13:2181
> > > -mesos_master_address=zk://192.168.0.13:2181/mesos/master
> > > -serverset_path=/aurora/scheduler
> > > -native_log_zk_group_path=/aurora/replicated-log
> > > -native_log_file_path=/usr/local/aurora-scheduler/db
> > > -backup_dir=/usr/local/aurora-scheduler/backups
> > > -thermos_executor_path=/usr/local/bin/thermos_executor
> > > -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
> > > jvm_prop_sun_java_launcher SUN_STANDARD
> > > jvm_prop_sun_jnu_encoding UTF-8
> > > jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
> > > jvm_prop_sun_os_patch_level unknown
> > > jvm_prop_user_country US
> > > jvm_prop_user_dir /home/lychee
> > > jvm_prop_user_home /root
> > > jvm_prop_user_language en
> > > jvm_prop_user_name root
> > > jvm_prop_user_timezone Etc/UTC
> > > jvm_threads_active 21
> > > jvm_threads_daemon 12
> > > jvm_threads_peak 25
> > > jvm_threads_started 867
> > > jvm_time_ms 1387559947214
> > > jvm_uptime_secs 38781
> > > log_compressed_entry_bytes_saved 2537635
> > > logging_dir /root
> > > mem_storage_consistent_read_operation_events 619
> > > mem_storage_consistent_read_operation_events_per_sec 0.0
> > > mem_storage_consistent_read_operation_nanos_per_event 0.0
> > > mem_storage_consistent_read_operation_nanos_total 272565131
> > > mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
> > > mem_storage_delete_all_tasks_events 1
> > > mem_storage_delete_all_tasks_events_per_sec 0.0
> > > mem_storage_delete_all_tasks_nanos_per_event 0.0
> > > mem_storage_delete_all_tasks_nanos_total 2323293
> > > mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
> > > mem_storage_delete_tasks_events 179
> > > mem_storage_delete_tasks_events_per_sec 0.0
> > > mem_storage_delete_tasks_nanos_per_event 0.0
> > > mem_storage_delete_tasks_nanos_total 8638495
> > > mem_storage_delete_tasks_nanos_total_per_sec 0.0
> > > mem_storage_fetch_tasks_events 3219
> > > mem_storage_fetch_tasks_events_per_sec 0.0
> > > mem_storage_fetch_tasks_nanos_per_event 0.0
> > > mem_storage_fetch_tasks_nanos_total 193613049
> > > mem_storage_fetch_tasks_nanos_total_per_sec 0.0
> > > mem_storage_mutate_tasks_events 677
> > > mem_storage_mutate_tasks_events_per_sec 0.0
> > > mem_storage_mutate_tasks_nanos_per_event 0.0
> > > mem_storage_mutate_tasks_nanos_total 149943165
> > > mem_storage_mutate_tasks_nanos_total_per_sec 0.0
> > > mem_storage_save_tasks_events 306
> > > mem_storage_save_tasks_events_per_sec 0.0
> > > mem_storage_save_tasks_nanos_per_event 0.0
> > > mem_storage_save_tasks_nanos_total 165988968
> > > mem_storage_save_tasks_nanos_total_per_sec 0.0
> > > mem_storage_weakly_consistent_read_operation_events 1538
> > > mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
> > > mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
> > > mem_storage_weakly_consistent_read_operation_nanos_total 138584844
> > > mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
> > > mem_storage_write_operation_events 2932
> > > mem_storage_write_operation_events_per_sec 0.0
> > > mem_storage_write_operation_nanos_per_event 0.0
> > > mem_storage_write_operation_nanos_total 28077154879
> > > mem_storage_write_operation_nanos_total_per_sec 0.0
> > > new_instance_id_set 0
> > > offer_accept_races 0
> > > old_instance_id_set 0
> > > outstanding_offers 1
> > > preemptor_attempts 0
> > > preemptor_failed_preemptions 0
> > > preemptor_no_slots_found 0
> > > preemptor_tasks_preempted 0
> > > process_cpu_cores_utilized 0.0
> > > process_cpu_time_nanos 73230000000
> > > process_max_fd_count 4096
> > > process_open_fd_count 157
> > > read_lock_wait_nanos 545041
> > > resources_allocated_quota_cpu 0
> > > resources_allocated_quota_disk_gb 0
> > > resources_allocated_quota_ram_gb 0
> > > resources_dedicated_consumed_cpu 0
> > > resources_dedicated_consumed_disk_gb 0
> > > resources_dedicated_consumed_ram_gb 0
> > > resources_free_pool_consumed_cpu 1
> > > resources_free_pool_consumed_disk_gb 0
> > > resources_free_pool_consumed_ram_gb 0
> > > resources_quota_consumed_cpu 0
> > > resources_quota_consumed_disk_gb 0
> > > resources_quota_consumed_ram_gb 0
> > > resources_total_consumed_cpu 1
> > > resources_total_consumed_disk_gb 0
> > > resources_total_consumed_ram_gb 0
> > > schedule_attempts_failed 0
> > > schedule_attempts_fired 135
> > > schedule_queue_size 1
> > > scheduler_backup_failed 0
> > > scheduler_backup_success 10
> > > scheduler_driver_kill_failures 0
> > > scheduler_failed_offers 0
> > > scheduler_framework_disconnects 0
> > > scheduler_framework_reregisters 0
> > > scheduler_illegal_task_state_transitions 200
> > > scheduler_log_bad_frames_read 0
> > > scheduler_log_bytes_read 292813
> > > scheduler_log_bytes_written 1080061
> > > scheduler_log_deflated_entries_read 1
> > > scheduler_log_entries_read 175
> > > scheduler_log_entries_written 555
> > > scheduler_log_native_append_events 556
> > > scheduler_log_native_append_events_per_sec 0.0
> > > scheduler_log_native_append_failures 0
> > > scheduler_log_native_append_nanos_per_event 0.0
> > > scheduler_log_native_append_nanos_total 25082628348
> > > scheduler_log_native_append_nanos_total_per_sec 0.0
> > > scheduler_log_native_append_timeouts 0
> > > scheduler_log_native_native_entries_skipped 1
> > > scheduler_log_native_read_events 176
> > > scheduler_log_native_read_events_per_sec 0.0
> > > scheduler_log_native_read_failures 0
> > > scheduler_log_native_read_nanos_per_event 0.0
> > > scheduler_log_native_read_nanos_total 73255842
> > > scheduler_log_native_read_nanos_total_per_sec 0.0
> > > scheduler_log_native_read_timeouts 0
> > > scheduler_log_native_truncate_events 10
> > > scheduler_log_native_truncate_events_per_sec 0.0
> > > scheduler_log_native_truncate_failures 0
> > > scheduler_log_native_truncate_nanos_per_event 0.0
> > > scheduler_log_native_truncate_nanos_total 276600750
> > > scheduler_log_native_truncate_nanos_total_per_sec 0.0
> > > scheduler_log_native_truncate_timeouts 0
> > > scheduler_log_recover_events 1
> > > scheduler_log_recover_events_per_sec 0.0
> > > scheduler_log_recover_nanos_per_event 0.0
> > > scheduler_log_recover_nanos_total 565333612
> > > scheduler_log_recover_nanos_total_per_sec 0.0
> > > scheduler_log_save_framework_id_events 2
> > > scheduler_log_save_framework_id_events_per_sec 0.0
> > > scheduler_log_save_framework_id_nanos_per_event 0.0
> > > scheduler_log_save_framework_id_nanos_total 437292
> > > scheduler_log_save_framework_id_nanos_total_per_sec 0.0
> > > scheduler_log_snapshot_events 10
> > > scheduler_log_snapshot_events_per_sec 0.0
> > > scheduler_log_snapshot_nanos_per_event 0.0
> > > scheduler_log_snapshot_nanos_total 961465418
> > > scheduler_log_snapshot_nanos_total_per_sec 0.0
> > > scheduler_log_snapshot_persist_events 10
> > > scheduler_log_snapshot_persist_events_per_sec 0.0
> > > scheduler_log_snapshot_persist_nanos_per_event 0.0
> > > scheduler_log_snapshot_persist_nanos_total 840444765
> > > scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
> > > scheduler_log_snapshots 10
> > > scheduler_log_tasks_mutate_events 677
> > > scheduler_log_tasks_mutate_events_per_sec 0.0
> > > scheduler_log_tasks_mutate_nanos_per_event 0.0
> > > scheduler_log_tasks_mutate_nanos_total 182346727
> > > scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
> > > scheduler_log_tasks_remove_events 179
> > > scheduler_log_tasks_remove_events_per_sec 0.0
> > > scheduler_log_tasks_remove_nanos_per_event 0.0
> > > scheduler_log_tasks_remove_nanos_total 13588163
> > > scheduler_log_tasks_remove_nanos_total_per_sec 0.0
> > > scheduler_log_tasks_save_events 305
> > > scheduler_log_tasks_save_events_per_sec 0.0
> > > scheduler_log_tasks_save_nanos_per_event 0.0
> > > scheduler_log_tasks_save_nanos_total 114479852
> > > scheduler_log_tasks_save_nanos_total_per_sec 0.0
> > > scheduler_log_un_snapshotted_transactions 36
> > > scheduler_lost_executors 0
> > > scheduler_max_ASSIGNED_waiting_ms 0
> > > scheduler_max_KILLING_waiting_ms 0
> > > scheduler_max_PREEMPTING_waiting_ms 0
> > > scheduler_max_RESTARTING_waiting_ms 0
> > > scheduler_resource_offers 397
> > > scheduler_resource_offers_events 397
> > > scheduler_resource_offers_events_per_sec 0.0
> > > scheduler_resource_offers_nanos_per_event 0.0
> > > scheduler_resource_offers_nanos_total 305282357
> > > scheduler_resource_offers_nanos_total_per_sec 0.0
> > > scheduler_save_host_attribute_events 397
> > > scheduler_save_host_attribute_events_per_sec 0.0
> > > scheduler_save_host_attribute_nanos_per_event 0.0
> > > scheduler_save_host_attribute_nanos_total 12466197
> > > scheduler_save_host_attribute_nanos_total_per_sec 0.0
> > > scheduler_status_update_events 426
> > > scheduler_status_update_events_per_sec 0.0
> > > scheduler_status_update_nanos_per_event 0.0
> > > scheduler_status_update_nanos_total 19334004116
> > > scheduler_status_update_nanos_total_per_sec 0.0
> > > scheduler_status_updates 0
> > > scheduler_thrift_createJob_events 2
> > > scheduler_thrift_createJob_events_per_sec 0.0
> > > scheduler_thrift_createJob_nanos_per_event 0.0
> > > scheduler_thrift_createJob_nanos_total 54102009
> > > scheduler_thrift_createJob_nanos_total_per_sec 0.0
> > > scheduler_thrift_getTasksStatus_events 2
> > > scheduler_thrift_getTasksStatus_events_per_sec 0.0
> > > scheduler_thrift_getTasksStatus_nanos_per_event 0.0
> > > scheduler_thrift_getTasksStatus_nanos_total 5684762
> > > scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
> > > scheduler_thrift_getVersion_events 4
> > > scheduler_thrift_getVersion_events_per_sec 0.0
> > > scheduler_thrift_getVersion_nanos_per_event 0.0
> > > scheduler_thrift_getVersion_nanos_total 13854714
> > > scheduler_thrift_getVersion_nanos_total_per_sec 0.0
> > > scheduler_thrift_killTasks_events 2
> > > scheduler_thrift_killTasks_events_per_sec 0.0
> > > scheduler_thrift_killTasks_nanos_per_event 0.0
> > > scheduler_thrift_killTasks_nanos_total 186407312
> > > scheduler_thrift_killTasks_nanos_total_per_sec 0.0
> > > shard_sanity_check_failures 0
> > > snapshot_apply_events 1
> > > snapshot_apply_events_per_sec 0.0
> > > snapshot_apply_nanos_per_event 0.0
> > > snapshot_apply_nanos_total 93051831
> > > snapshot_apply_nanos_total_per_sec 0.0
> > > snapshot_create_events 10
> > > snapshot_create_events_per_sec 0.0
> > > snapshot_create_nanos_per_event 0.0
> > > snapshot_create_nanos_total 37066472
> > > snapshot_create_nanos_total_per_sec 0.0
> > > system_env_GLOG_v 0
> > > system_env_HOME /home/lychee
> > > system_env_JAVA_OPTS -server -Xmx1g -Xms1g
> > > -Djava.library.path=/usr/local/lib
> > > system_env_LANG en_US.UTF-8
> > > system_env_LIBPROCESS_IP 192.168.0.18
> > > system_env_LIBPROCESS_PORT 8083
> > > system_env_LOGNAME root
> > > system_env_MAIL /var/mail/root
> > > system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> > > system_env_OLDPWD /usr/local/aurora-scheduler
> > > system_env_PATH
> > > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > > system_env_PWD /home/lychee
> > > system_env_SHELL /bin/bash
> > > system_env_SHLVL 0
> > > system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
> > > system_env_SUDO_GID 1002
> > > system_env_SUDO_UID 1002
> > > system_env_SUDO_USER lychee
> > > system_env_TERM xterm
> > > system_env_USER root
> > > system_env_USERNAME root
> > > system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
> > > system_free_physical_memory_mb 148
> > > system_free_swap_mb 8063
> > > system_load_avg 2.22
> > > task_queries_all 45
> > > task_queries_by_id 3668
> > > task_queries_by_job 183
> > > task_schedule_attempt_events 135
> > > task_schedule_attempt_events_per_sec 0.0
> > > task_schedule_attempt_nanos_per_event 0.0
> > > task_schedule_attempt_nanos_total 6289005821
> > > task_schedule_attempt_nanos_total_per_sec 0.0
> > > task_store_ASSIGNED 0
> > > task_store_FAILED 0
> > > task_store_FINISHED 200
> > > task_store_INIT 0
> > > task_store_KILLED 0
> > > task_store_KILLING 0
> > > task_store_LOST 0
> > > task_store_PENDING 1
> > > task_store_PREEMPTING 0
> > > task_store_RESTARTING 0
> > > task_store_RUNNING 0
> > > task_store_STARTING 0
> > > task_store_THROTTLED 0
> > > task_store_UNKNOWN 0
> > > timed_out_tasks 0
> > > timeout_queue_size 338
> > > transient_states 0
> > > uncaught_exceptions 0
> > > variable_scrape_events 38778
> > > variable_scrape_events_per_sec 1.0000103021061324
> > > variable_scrape_micros_per_event 455.0001369564521
> > > variable_scrape_micros_total 18809481
> > > variable_scrape_micros_total_per_sec 455.00482441615327
> > > work_queue_depth 0
> > > write_lock_wait_nanos 13138744
> > >
> > >
> > >
> > >
> > > On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
> > > >wrote:
> > >
> > > > Hi Yingbo,
> > > >
> > > > Do you mind sharing your .aurora file? Perhaps your are requesting
> > > > Resources or Constraints that are infeasible for your local setup to
> > > > satisfy?
> > > >
> > > > Also, you might want to check that your scheduler is receiving offers
> > > from
> > > > mesos master. You could check your scheduler log file to see if any
> > > offers
> > > > are coming. Alternatively, you might want to look at
> > > > "scheduler_resource_offers" stat exposed by the scheduler /vars
> > endpoint
> > > > (http://<your_local_scheduler_ip:port>/vars).
> > > >
> > > > Thanks,
> > > > Maxim
> > > >
> > > >
> > > >
> > > > On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> > > >
> > > > > hi,
> > > > > Hope you guys have a fun night at city hall.
> > > > > While you might be drunk, can anyone help me a little bit on
> > setting
> > > > up
> > > > > the local dev environment to debug aurora ?
> > > > > My current set up.
> > > > > Machine A: 64bit ubuntu server running mesos-master,
> > > > > Machine B: 64bit ubuntu server running mesos-slave, thermo
> > > > > observer server;
> > > > > Machine C: 32bit ubuntu server running zookeeper.
> > > > > Machine D: Mac air for developing aurora.
> > > > >
> > > > > I tried to run aurora-scheduler on Machine A and also created
> a
> > > > hello
> > > > > job from machine A. the job was submitted successfully. however the
> > job
> > > > > keeps staying at "PENDING" status. i checked thermo observer from
> > port
> > > > > 1336, it says "No matching hosts".
> > > > >
> > > > > I spent a couple of hours and still can't figure out the
> > reason
> > > so
> > > > > I decided to settup the dev environment on my Mac so that I can
> trace
> > > the
> > > > > error in IntelliJ. Can someone give me some hint on that? I am not
> > > quite
> > > > > familiar with IntelliJ and java.
> > > > >
> > > > >
> > > > >
> > > > > best
> > > > > Yingbo
> > > >
> > > >
> > >
> >
>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Yingbo Wang <yb...@gmail.com>.
thanks Maxim. I think that would resolve the issue I have right now.
best
Yingbo
On Fri, Dec 20, 2013 at 12:23 PM, Maxim Khutornenko <ma...@twitter.com>wrote:
> Yingbo,
>
> Since your job was defined as service but was exiting right away, it was
> penalized by the scheduler for its a "flapping" behavior. The penalty is
> set by the exponential back-off with a max cut off at 5 minutes. The "No
> matching hosts" displayed in the UI is a bit misleading here and will be
> improved soon when the Throttled task state is implemented.
>
> Thanks,
> Maxim
>
>
>
> On Dec 20, 2013, at 12:01 PM, Yingbo Wang <yb...@gmail.com> wrote:
>
> > It is a good catch! Thanks Brian.
> >
> > The task indeed restarts every 5 minutes. I guess the job just doesn't
> > can't find the running service that's why it says "Not matching hosts"?
> > I will try it to see if it works.
> >
> >
> >
> >
> > On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman <wickman@twopensource.com
> >wrote:
> >
> >> There is an executor_id in the state there, so it looks like it is
> managing
> >> to launch the tasks. Since you've listed your job as a Service(), it
> means
> >> that every task will be rescheduled after it exits, and since it looks
> like
> >> your task just immediately returns, it's probably launching dozens of
> them
> >> per minute if not more. You can change this behavior by changing
> Service
> >> to Job. (Service is just an alias for Job(service=True).)
> >>
> >> ~brian
> >>
> >>
> >>
> >> On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
> >>
> >>> Thanks Maxim.
> >>>
> >>> did you mean hello.aurora?
> >>>
> >>>
> >>> hello = Process(
> >>> name = 'hello',
> >>> cmdline = """
> >>> echo "ls file " ;
> >>> whoami;
> >>> hostname;
> >>> """)
> >>>
> >>> task = SequentialTask(
> >>> processes = [hello],
> >>> resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
> >>>
> >>> jobs = [Service(
> >>> task = task, cluster = 'lychee', role = 'lychee', environment =
> 'prod',
> >>> name = 'hello')]
> >>>
> >>> The slave machine has enough resource to satisfy this request.
> >>>
> >>> I did encounter an issue saying "Constraint Not Satisfied, host".
> After I
> >>> change my hostname of both master and slave machines, this problem
> seems
> >> to
> >>> be solved, and now the problem becomes "No matching hosts"
> >>>
> >>>
> >>>
> >>> Here's the offers that scheduler received:
> >>>
> >>> [
> >>>
> >>> -
> >>> {
> >>> - id: "201312200632-302033088-5050-15916-396",
> >>> - framework_id: "201312110654-302033088-5050-31688-0000",
> >>> - slave_id: "201312170824-302033088-5050-25318-0",
> >>> - hostname: "192.168.0.11",
> >>> - resources:
> >>> [
> >>> -
> >>> {
> >>> - name: "cpus",
> >>> - scalar: 7.75
> >>> },
> >>> -
> >>> {
> >>> - name: "disk",
> >>> - scalar: 917136
> >>> },
> >>> -
> >>> {
> >>> - name: "mem",
> >>> - scalar: 14814
> >>> },
> >>> -
> >>> {
> >>> - name: "ports",
> >>> - ranges:
> >>> [
> >>> - "31000-32000"
> >>> ]
> >>> }
> >>> ],
> >>> - attributes:
> >>> [
> >>> -
> >>> {
> >>> - name: "host",
> >>> - text: "192.168.0.11"
> >>> },
> >>> -
> >>> {
> >>> - name: "rack",
> >>> - text: "a"
> >>> }
> >>> ],
> >>> - executor_ids:
> >>> [
> >>> -
> >>>
> >>>
> >>
> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
> >>> ]
> >>> }
> >>>
> >>> ]
> >>>
> >>>
> >>> and the /vars are:
> >>>
> >>>
> >>> async_tasks_completed 393
> >>> both_instance_ids_set 0
> >>> build_date null
> >>> build_git_branchname null
> >>> build_git_revision null
> >>> build_git_revision_number null
> >>> build_git_tag null
> >>> build_machine null
> >>> build_path null
> >>> build_time null
> >>> build_timestamp null
> >>> build_user_name null
> >>> cron_job_launch_failures 0
> >>> cron_jobs_triggered 0
> >>> cron_num_pending_runs 0
> >>> empty_slots_large 0
> >>> empty_slots_medium 1
> >>> empty_slots_small 7
> >>> empty_slots_xlarge 0
> >>> framework_registered 1
> >>> http_200_responses_events 22
> >>> http_200_responses_events_per_sec 0.0
> >>> http_200_responses_nanos_per_event 0.0
> >>> http_200_responses_nanos_total 981542679
> >>> http_200_responses_nanos_total_per_sec 0.0
> >>> instance_ids_inconsistent 0
> >>> jvm_available_processors 4
> >>> jvm_class_loaded_count 6658
> >>> jvm_class_total_loaded_count 6658
> >>> jvm_class_unloaded_count 0
> >>> jvm_gc_PS_MarkSweep_collection_count 0
> >>> jvm_gc_PS_MarkSweep_collection_time_ms 0
> >>> jvm_gc_PS_Scavenge_collection_count 7
> >>> jvm_gc_PS_Scavenge_collection_time_ms 534
> >>> jvm_gc_collection_count 7
> >>> jvm_gc_collection_time_ms 534
> >>> jvm_input_arguments [-Xmx1g, -Xms1g,
> -Djava.library.path=/usr/local/lib]
> >>> jvm_memory_free_mb 688
> >>> jvm_memory_heap_mb_committed 968
> >>> jvm_memory_heap_mb_max 968
> >>> jvm_memory_heap_mb_used 279
> >>> jvm_memory_max_mb 968
> >>> jvm_memory_mb_total 968
> >>> jvm_memory_non_heap_mb_committed 39
> >>> jvm_memory_non_heap_mb_max 130
> >>> jvm_memory_non_heap_mb_used 38
> >>> jvm_prop_awt_toolkit sun.awt.X11.XToolkit
> >>> jvm_prop_file_encoding UTF-8
> >>> jvm_prop_file_encoding_pkg sun.io
> >>> jvm_prop_file_separator /
> >>> jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
> >>> jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
> >>> jvm_prop_java_class_path
> >>>
> >>>
> >>
> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
> >>> jvm_prop_java_class_version 51.0
> >>> jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
> >>> jvm_prop_java_ext_dirs
> >>> /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
> >>> jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
> >>> jvm_prop_java_io_tmpdir /tmp
> >>> jvm_prop_java_library_path /usr/local/lib
> >>> jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
> >>> jvm_prop_java_runtime_version 1.7.0_45-b18
> >>> jvm_prop_java_specification_name Java Platform API Specification
> >>> jvm_prop_java_specification_vendor Oracle Corporation
> >>> jvm_prop_java_specification_version 1.7
> >>> jvm_prop_java_vendor Oracle Corporation
> >>> jvm_prop_java_vendor_url http://java.oracle.com/
> >>> jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
> >>> jvm_prop_java_version 1.7.0_45
> >>> jvm_prop_java_vm_info mixed mode
> >>> jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
> >>> jvm_prop_java_vm_specification_name Java Virtual Machine Specification
> >>> jvm_prop_java_vm_specification_vendor Oracle Corporation
> >>> jvm_prop_java_vm_specification_version 1.7
> >>> jvm_prop_java_vm_vendor Oracle Corporation
> >>> jvm_prop_java_vm_version 24.45-b08
> >>> jvm_prop_line_separator
> >>>
> >>> jvm_prop_os_arch amd64
> >>> jvm_prop_os_name Linux
> >>> jvm_prop_os_version 3.5.0-44-generic
> >>> jvm_prop_path_separator :
> >>> jvm_prop_sun_arch_data_model 64
> >>> jvm_prop_sun_boot_class_path
> >>>
> >>>
> >>
> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
> >>> jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
> >>> jvm_prop_sun_cpu_endian little
> >>> jvm_prop_sun_cpu_isalist
> >>> jvm_prop_sun_io_unicode_encoding UnicodeLittle
> >>> jvm_prop_sun_java_command
> >>> com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
> >>> -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
> >>> -zk_endpoints=192.168.0.13:2181
> >>> -mesos_master_address=zk://192.168.0.13:2181/mesos/master
> >>> -serverset_path=/aurora/scheduler
> >>> -native_log_zk_group_path=/aurora/replicated-log
> >>> -native_log_file_path=/usr/local/aurora-scheduler/db
> >>> -backup_dir=/usr/local/aurora-scheduler/backups
> >>> -thermos_executor_path=/usr/local/bin/thermos_executor
> >>> -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
> >>> jvm_prop_sun_java_launcher SUN_STANDARD
> >>> jvm_prop_sun_jnu_encoding UTF-8
> >>> jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
> >>> jvm_prop_sun_os_patch_level unknown
> >>> jvm_prop_user_country US
> >>> jvm_prop_user_dir /home/lychee
> >>> jvm_prop_user_home /root
> >>> jvm_prop_user_language en
> >>> jvm_prop_user_name root
> >>> jvm_prop_user_timezone Etc/UTC
> >>> jvm_threads_active 21
> >>> jvm_threads_daemon 12
> >>> jvm_threads_peak 25
> >>> jvm_threads_started 867
> >>> jvm_time_ms 1387559947214
> >>> jvm_uptime_secs 38781
> >>> log_compressed_entry_bytes_saved 2537635
> >>> logging_dir /root
> >>> mem_storage_consistent_read_operation_events 619
> >>> mem_storage_consistent_read_operation_events_per_sec 0.0
> >>> mem_storage_consistent_read_operation_nanos_per_event 0.0
> >>> mem_storage_consistent_read_operation_nanos_total 272565131
> >>> mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
> >>> mem_storage_delete_all_tasks_events 1
> >>> mem_storage_delete_all_tasks_events_per_sec 0.0
> >>> mem_storage_delete_all_tasks_nanos_per_event 0.0
> >>> mem_storage_delete_all_tasks_nanos_total 2323293
> >>> mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
> >>> mem_storage_delete_tasks_events 179
> >>> mem_storage_delete_tasks_events_per_sec 0.0
> >>> mem_storage_delete_tasks_nanos_per_event 0.0
> >>> mem_storage_delete_tasks_nanos_total 8638495
> >>> mem_storage_delete_tasks_nanos_total_per_sec 0.0
> >>> mem_storage_fetch_tasks_events 3219
> >>> mem_storage_fetch_tasks_events_per_sec 0.0
> >>> mem_storage_fetch_tasks_nanos_per_event 0.0
> >>> mem_storage_fetch_tasks_nanos_total 193613049
> >>> mem_storage_fetch_tasks_nanos_total_per_sec 0.0
> >>> mem_storage_mutate_tasks_events 677
> >>> mem_storage_mutate_tasks_events_per_sec 0.0
> >>> mem_storage_mutate_tasks_nanos_per_event 0.0
> >>> mem_storage_mutate_tasks_nanos_total 149943165
> >>> mem_storage_mutate_tasks_nanos_total_per_sec 0.0
> >>> mem_storage_save_tasks_events 306
> >>> mem_storage_save_tasks_events_per_sec 0.0
> >>> mem_storage_save_tasks_nanos_per_event 0.0
> >>> mem_storage_save_tasks_nanos_total 165988968
> >>> mem_storage_save_tasks_nanos_total_per_sec 0.0
> >>> mem_storage_weakly_consistent_read_operation_events 1538
> >>> mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
> >>> mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
> >>> mem_storage_weakly_consistent_read_operation_nanos_total 138584844
> >>> mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
> >>> mem_storage_write_operation_events 2932
> >>> mem_storage_write_operation_events_per_sec 0.0
> >>> mem_storage_write_operation_nanos_per_event 0.0
> >>> mem_storage_write_operation_nanos_total 28077154879
> >>> mem_storage_write_operation_nanos_total_per_sec 0.0
> >>> new_instance_id_set 0
> >>> offer_accept_races 0
> >>> old_instance_id_set 0
> >>> outstanding_offers 1
> >>> preemptor_attempts 0
> >>> preemptor_failed_preemptions 0
> >>> preemptor_no_slots_found 0
> >>> preemptor_tasks_preempted 0
> >>> process_cpu_cores_utilized 0.0
> >>> process_cpu_time_nanos 73230000000
> >>> process_max_fd_count 4096
> >>> process_open_fd_count 157
> >>> read_lock_wait_nanos 545041
> >>> resources_allocated_quota_cpu 0
> >>> resources_allocated_quota_disk_gb 0
> >>> resources_allocated_quota_ram_gb 0
> >>> resources_dedicated_consumed_cpu 0
> >>> resources_dedicated_consumed_disk_gb 0
> >>> resources_dedicated_consumed_ram_gb 0
> >>> resources_free_pool_consumed_cpu 1
> >>> resources_free_pool_consumed_disk_gb 0
> >>> resources_free_pool_consumed_ram_gb 0
> >>> resources_quota_consumed_cpu 0
> >>> resources_quota_consumed_disk_gb 0
> >>> resources_quota_consumed_ram_gb 0
> >>> resources_total_consumed_cpu 1
> >>> resources_total_consumed_disk_gb 0
> >>> resources_total_consumed_ram_gb 0
> >>> schedule_attempts_failed 0
> >>> schedule_attempts_fired 135
> >>> schedule_queue_size 1
> >>> scheduler_backup_failed 0
> >>> scheduler_backup_success 10
> >>> scheduler_driver_kill_failures 0
> >>> scheduler_failed_offers 0
> >>> scheduler_framework_disconnects 0
> >>> scheduler_framework_reregisters 0
> >>> scheduler_illegal_task_state_transitions 200
> >>> scheduler_log_bad_frames_read 0
> >>> scheduler_log_bytes_read 292813
> >>> scheduler_log_bytes_written 1080061
> >>> scheduler_log_deflated_entries_read 1
> >>> scheduler_log_entries_read 175
> >>> scheduler_log_entries_written 555
> >>> scheduler_log_native_append_events 556
> >>> scheduler_log_native_append_events_per_sec 0.0
> >>> scheduler_log_native_append_failures 0
> >>> scheduler_log_native_append_nanos_per_event 0.0
> >>> scheduler_log_native_append_nanos_total 25082628348
> >>> scheduler_log_native_append_nanos_total_per_sec 0.0
> >>> scheduler_log_native_append_timeouts 0
> >>> scheduler_log_native_native_entries_skipped 1
> >>> scheduler_log_native_read_events 176
> >>> scheduler_log_native_read_events_per_sec 0.0
> >>> scheduler_log_native_read_failures 0
> >>> scheduler_log_native_read_nanos_per_event 0.0
> >>> scheduler_log_native_read_nanos_total 73255842
> >>> scheduler_log_native_read_nanos_total_per_sec 0.0
> >>> scheduler_log_native_read_timeouts 0
> >>> scheduler_log_native_truncate_events 10
> >>> scheduler_log_native_truncate_events_per_sec 0.0
> >>> scheduler_log_native_truncate_failures 0
> >>> scheduler_log_native_truncate_nanos_per_event 0.0
> >>> scheduler_log_native_truncate_nanos_total 276600750
> >>> scheduler_log_native_truncate_nanos_total_per_sec 0.0
> >>> scheduler_log_native_truncate_timeouts 0
> >>> scheduler_log_recover_events 1
> >>> scheduler_log_recover_events_per_sec 0.0
> >>> scheduler_log_recover_nanos_per_event 0.0
> >>> scheduler_log_recover_nanos_total 565333612
> >>> scheduler_log_recover_nanos_total_per_sec 0.0
> >>> scheduler_log_save_framework_id_events 2
> >>> scheduler_log_save_framework_id_events_per_sec 0.0
> >>> scheduler_log_save_framework_id_nanos_per_event 0.0
> >>> scheduler_log_save_framework_id_nanos_total 437292
> >>> scheduler_log_save_framework_id_nanos_total_per_sec 0.0
> >>> scheduler_log_snapshot_events 10
> >>> scheduler_log_snapshot_events_per_sec 0.0
> >>> scheduler_log_snapshot_nanos_per_event 0.0
> >>> scheduler_log_snapshot_nanos_total 961465418
> >>> scheduler_log_snapshot_nanos_total_per_sec 0.0
> >>> scheduler_log_snapshot_persist_events 10
> >>> scheduler_log_snapshot_persist_events_per_sec 0.0
> >>> scheduler_log_snapshot_persist_nanos_per_event 0.0
> >>> scheduler_log_snapshot_persist_nanos_total 840444765
> >>> scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
> >>> scheduler_log_snapshots 10
> >>> scheduler_log_tasks_mutate_events 677
> >>> scheduler_log_tasks_mutate_events_per_sec 0.0
> >>> scheduler_log_tasks_mutate_nanos_per_event 0.0
> >>> scheduler_log_tasks_mutate_nanos_total 182346727
> >>> scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
> >>> scheduler_log_tasks_remove_events 179
> >>> scheduler_log_tasks_remove_events_per_sec 0.0
> >>> scheduler_log_tasks_remove_nanos_per_event 0.0
> >>> scheduler_log_tasks_remove_nanos_total 13588163
> >>> scheduler_log_tasks_remove_nanos_total_per_sec 0.0
> >>> scheduler_log_tasks_save_events 305
> >>> scheduler_log_tasks_save_events_per_sec 0.0
> >>> scheduler_log_tasks_save_nanos_per_event 0.0
> >>> scheduler_log_tasks_save_nanos_total 114479852
> >>> scheduler_log_tasks_save_nanos_total_per_sec 0.0
> >>> scheduler_log_un_snapshotted_transactions 36
> >>> scheduler_lost_executors 0
> >>> scheduler_max_ASSIGNED_waiting_ms 0
> >>> scheduler_max_KILLING_waiting_ms 0
> >>> scheduler_max_PREEMPTING_waiting_ms 0
> >>> scheduler_max_RESTARTING_waiting_ms 0
> >>> scheduler_resource_offers 397
> >>> scheduler_resource_offers_events 397
> >>> scheduler_resource_offers_events_per_sec 0.0
> >>> scheduler_resource_offers_nanos_per_event 0.0
> >>> scheduler_resource_offers_nanos_total 305282357
> >>> scheduler_resource_offers_nanos_total_per_sec 0.0
> >>> scheduler_save_host_attribute_events 397
> >>> scheduler_save_host_attribute_events_per_sec 0.0
> >>> scheduler_save_host_attribute_nanos_per_event 0.0
> >>> scheduler_save_host_attribute_nanos_total 12466197
> >>> scheduler_save_host_attribute_nanos_total_per_sec 0.0
> >>> scheduler_status_update_events 426
> >>> scheduler_status_update_events_per_sec 0.0
> >>> scheduler_status_update_nanos_per_event 0.0
> >>> scheduler_status_update_nanos_total 19334004116
> >>> scheduler_status_update_nanos_total_per_sec 0.0
> >>> scheduler_status_updates 0
> >>> scheduler_thrift_createJob_events 2
> >>> scheduler_thrift_createJob_events_per_sec 0.0
> >>> scheduler_thrift_createJob_nanos_per_event 0.0
> >>> scheduler_thrift_createJob_nanos_total 54102009
> >>> scheduler_thrift_createJob_nanos_total_per_sec 0.0
> >>> scheduler_thrift_getTasksStatus_events 2
> >>> scheduler_thrift_getTasksStatus_events_per_sec 0.0
> >>> scheduler_thrift_getTasksStatus_nanos_per_event 0.0
> >>> scheduler_thrift_getTasksStatus_nanos_total 5684762
> >>> scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
> >>> scheduler_thrift_getVersion_events 4
> >>> scheduler_thrift_getVersion_events_per_sec 0.0
> >>> scheduler_thrift_getVersion_nanos_per_event 0.0
> >>> scheduler_thrift_getVersion_nanos_total 13854714
> >>> scheduler_thrift_getVersion_nanos_total_per_sec 0.0
> >>> scheduler_thrift_killTasks_events 2
> >>> scheduler_thrift_killTasks_events_per_sec 0.0
> >>> scheduler_thrift_killTasks_nanos_per_event 0.0
> >>> scheduler_thrift_killTasks_nanos_total 186407312
> >>> scheduler_thrift_killTasks_nanos_total_per_sec 0.0
> >>> shard_sanity_check_failures 0
> >>> snapshot_apply_events 1
> >>> snapshot_apply_events_per_sec 0.0
> >>> snapshot_apply_nanos_per_event 0.0
> >>> snapshot_apply_nanos_total 93051831
> >>> snapshot_apply_nanos_total_per_sec 0.0
> >>> snapshot_create_events 10
> >>> snapshot_create_events_per_sec 0.0
> >>> snapshot_create_nanos_per_event 0.0
> >>> snapshot_create_nanos_total 37066472
> >>> snapshot_create_nanos_total_per_sec 0.0
> >>> system_env_GLOG_v 0
> >>> system_env_HOME /home/lychee
> >>> system_env_JAVA_OPTS -server -Xmx1g -Xms1g
> >>> -Djava.library.path=/usr/local/lib
> >>> system_env_LANG en_US.UTF-8
> >>> system_env_LIBPROCESS_IP 192.168.0.18
> >>> system_env_LIBPROCESS_PORT 8083
> >>> system_env_LOGNAME root
> >>> system_env_MAIL /var/mail/root
> >>> system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> >>> system_env_OLDPWD /usr/local/aurora-scheduler
> >>> system_env_PATH
> >>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> >>> system_env_PWD /home/lychee
> >>> system_env_SHELL /bin/bash
> >>> system_env_SHLVL 0
> >>> system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
> >>> system_env_SUDO_GID 1002
> >>> system_env_SUDO_UID 1002
> >>> system_env_SUDO_USER lychee
> >>> system_env_TERM xterm
> >>> system_env_USER root
> >>> system_env_USERNAME root
> >>> system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
> >>> system_free_physical_memory_mb 148
> >>> system_free_swap_mb 8063
> >>> system_load_avg 2.22
> >>> task_queries_all 45
> >>> task_queries_by_id 3668
> >>> task_queries_by_job 183
> >>> task_schedule_attempt_events 135
> >>> task_schedule_attempt_events_per_sec 0.0
> >>> task_schedule_attempt_nanos_per_event 0.0
> >>> task_schedule_attempt_nanos_total 6289005821
> >>> task_schedule_attempt_nanos_total_per_sec 0.0
> >>> task_store_ASSIGNED 0
> >>> task_store_FAILED 0
> >>> task_store_FINISHED 200
> >>> task_store_INIT 0
> >>> task_store_KILLED 0
> >>> task_store_KILLING 0
> >>> task_store_LOST 0
> >>> task_store_PENDING 1
> >>> task_store_PREEMPTING 0
> >>> task_store_RESTARTING 0
> >>> task_store_RUNNING 0
> >>> task_store_STARTING 0
> >>> task_store_THROTTLED 0
> >>> task_store_UNKNOWN 0
> >>> timed_out_tasks 0
> >>> timeout_queue_size 338
> >>> transient_states 0
> >>> uncaught_exceptions 0
> >>> variable_scrape_events 38778
> >>> variable_scrape_events_per_sec 1.0000103021061324
> >>> variable_scrape_micros_per_event 455.0001369564521
> >>> variable_scrape_micros_total 18809481
> >>> variable_scrape_micros_total_per_sec 455.00482441615327
> >>> work_queue_depth 0
> >>> write_lock_wait_nanos 13138744
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
> >>>> wrote:
> >>>
> >>>> Hi Yingbo,
> >>>>
> >>>> Do you mind sharing your .aurora file? Perhaps your are requesting
> >>>> Resources or Constraints that are infeasible for your local setup to
> >>>> satisfy?
> >>>>
> >>>> Also, you might want to check that your scheduler is receiving offers
> >>> from
> >>>> mesos master. You could check your scheduler log file to see if any
> >>> offers
> >>>> are coming. Alternatively, you might want to look at
> >>>> "scheduler_resource_offers" stat exposed by the scheduler /vars
> >> endpoint
> >>>> (http://<your_local_scheduler_ip:port>/vars).
> >>>>
> >>>> Thanks,
> >>>> Maxim
> >>>>
> >>>>
> >>>>
> >>>> On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> >>>>
> >>>>> hi,
> >>>>> Hope you guys have a fun night at city hall.
> >>>>> While you might be drunk, can anyone help me a little bit on
> >> setting
> >>>> up
> >>>>> the local dev environment to debug aurora ?
> >>>>> My current set up.
> >>>>> Machine A: 64bit ubuntu server running mesos-master,
> >>>>> Machine B: 64bit ubuntu server running mesos-slave, thermo
> >>>>> observer server;
> >>>>> Machine C: 32bit ubuntu server running zookeeper.
> >>>>> Machine D: Mac air for developing aurora.
> >>>>>
> >>>>> I tried to run aurora-scheduler on Machine A and also created a
> >>>> hello
> >>>>> job from machine A. the job was submitted successfully. however the
> >> job
> >>>>> keeps staying at "PENDING" status. i checked thermo observer from
> >> port
> >>>>> 1336, it says "No matching hosts".
> >>>>>
> >>>>> I spent a couple of hours and still can't figure out the
> >> reason
> >>> so
> >>>>> I decided to settup the dev environment on my Mac so that I can trace
> >>> the
> >>>>> error in IntelliJ. Can someone give me some hint on that? I am not
> >>> quite
> >>>>> familiar with IntelliJ and java.
> >>>>>
> >>>>>
> >>>>>
> >>>>> best
> >>>>> Yingbo
> >>>>
> >>>>
> >>>
> >>
>
>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Maxim Khutornenko <ma...@twitter.com>.
Yingbo,
Since your job was defined as service but was exiting right away, it was penalized by the scheduler for its a "flapping" behavior. The penalty is set by the exponential back-off with a max cut off at 5 minutes. The "No matching hosts" displayed in the UI is a bit misleading here and will be improved soon when the Throttled task state is implemented.
Thanks,
Maxim
On Dec 20, 2013, at 12:01 PM, Yingbo Wang <yb...@gmail.com> wrote:
> It is a good catch! Thanks Brian.
>
> The task indeed restarts every 5 minutes. I guess the job just doesn't
> can't find the running service that's why it says "Not matching hosts"?
> I will try it to see if it works.
>
>
>
>
> On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman <wi...@twopensource.com>wrote:
>
>> There is an executor_id in the state there, so it looks like it is managing
>> to launch the tasks. Since you've listed your job as a Service(), it means
>> that every task will be rescheduled after it exits, and since it looks like
>> your task just immediately returns, it's probably launching dozens of them
>> per minute if not more. You can change this behavior by changing Service
>> to Job. (Service is just an alias for Job(service=True).)
>>
>> ~brian
>>
>>
>>
>> On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
>>
>>> Thanks Maxim.
>>>
>>> did you mean hello.aurora?
>>>
>>>
>>> hello = Process(
>>> name = 'hello',
>>> cmdline = """
>>> echo "ls file " ;
>>> whoami;
>>> hostname;
>>> """)
>>>
>>> task = SequentialTask(
>>> processes = [hello],
>>> resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
>>>
>>> jobs = [Service(
>>> task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
>>> name = 'hello')]
>>>
>>> The slave machine has enough resource to satisfy this request.
>>>
>>> I did encounter an issue saying "Constraint Not Satisfied, host". After I
>>> change my hostname of both master and slave machines, this problem seems
>> to
>>> be solved, and now the problem becomes "No matching hosts"
>>>
>>>
>>>
>>> Here's the offers that scheduler received:
>>>
>>> [
>>>
>>> -
>>> {
>>> - id: "201312200632-302033088-5050-15916-396",
>>> - framework_id: "201312110654-302033088-5050-31688-0000",
>>> - slave_id: "201312170824-302033088-5050-25318-0",
>>> - hostname: "192.168.0.11",
>>> - resources:
>>> [
>>> -
>>> {
>>> - name: "cpus",
>>> - scalar: 7.75
>>> },
>>> -
>>> {
>>> - name: "disk",
>>> - scalar: 917136
>>> },
>>> -
>>> {
>>> - name: "mem",
>>> - scalar: 14814
>>> },
>>> -
>>> {
>>> - name: "ports",
>>> - ranges:
>>> [
>>> - "31000-32000"
>>> ]
>>> }
>>> ],
>>> - attributes:
>>> [
>>> -
>>> {
>>> - name: "host",
>>> - text: "192.168.0.11"
>>> },
>>> -
>>> {
>>> - name: "rack",
>>> - text: "a"
>>> }
>>> ],
>>> - executor_ids:
>>> [
>>> -
>>>
>>>
>> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
>>> ]
>>> }
>>>
>>> ]
>>>
>>>
>>> and the /vars are:
>>>
>>>
>>> async_tasks_completed 393
>>> both_instance_ids_set 0
>>> build_date null
>>> build_git_branchname null
>>> build_git_revision null
>>> build_git_revision_number null
>>> build_git_tag null
>>> build_machine null
>>> build_path null
>>> build_time null
>>> build_timestamp null
>>> build_user_name null
>>> cron_job_launch_failures 0
>>> cron_jobs_triggered 0
>>> cron_num_pending_runs 0
>>> empty_slots_large 0
>>> empty_slots_medium 1
>>> empty_slots_small 7
>>> empty_slots_xlarge 0
>>> framework_registered 1
>>> http_200_responses_events 22
>>> http_200_responses_events_per_sec 0.0
>>> http_200_responses_nanos_per_event 0.0
>>> http_200_responses_nanos_total 981542679
>>> http_200_responses_nanos_total_per_sec 0.0
>>> instance_ids_inconsistent 0
>>> jvm_available_processors 4
>>> jvm_class_loaded_count 6658
>>> jvm_class_total_loaded_count 6658
>>> jvm_class_unloaded_count 0
>>> jvm_gc_PS_MarkSweep_collection_count 0
>>> jvm_gc_PS_MarkSweep_collection_time_ms 0
>>> jvm_gc_PS_Scavenge_collection_count 7
>>> jvm_gc_PS_Scavenge_collection_time_ms 534
>>> jvm_gc_collection_count 7
>>> jvm_gc_collection_time_ms 534
>>> jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
>>> jvm_memory_free_mb 688
>>> jvm_memory_heap_mb_committed 968
>>> jvm_memory_heap_mb_max 968
>>> jvm_memory_heap_mb_used 279
>>> jvm_memory_max_mb 968
>>> jvm_memory_mb_total 968
>>> jvm_memory_non_heap_mb_committed 39
>>> jvm_memory_non_heap_mb_max 130
>>> jvm_memory_non_heap_mb_used 38
>>> jvm_prop_awt_toolkit sun.awt.X11.XToolkit
>>> jvm_prop_file_encoding UTF-8
>>> jvm_prop_file_encoding_pkg sun.io
>>> jvm_prop_file_separator /
>>> jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
>>> jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
>>> jvm_prop_java_class_path
>>>
>>>
>> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
>>> jvm_prop_java_class_version 51.0
>>> jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
>>> jvm_prop_java_ext_dirs
>>> /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
>>> jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
>>> jvm_prop_java_io_tmpdir /tmp
>>> jvm_prop_java_library_path /usr/local/lib
>>> jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
>>> jvm_prop_java_runtime_version 1.7.0_45-b18
>>> jvm_prop_java_specification_name Java Platform API Specification
>>> jvm_prop_java_specification_vendor Oracle Corporation
>>> jvm_prop_java_specification_version 1.7
>>> jvm_prop_java_vendor Oracle Corporation
>>> jvm_prop_java_vendor_url http://java.oracle.com/
>>> jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
>>> jvm_prop_java_version 1.7.0_45
>>> jvm_prop_java_vm_info mixed mode
>>> jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
>>> jvm_prop_java_vm_specification_name Java Virtual Machine Specification
>>> jvm_prop_java_vm_specification_vendor Oracle Corporation
>>> jvm_prop_java_vm_specification_version 1.7
>>> jvm_prop_java_vm_vendor Oracle Corporation
>>> jvm_prop_java_vm_version 24.45-b08
>>> jvm_prop_line_separator
>>>
>>> jvm_prop_os_arch amd64
>>> jvm_prop_os_name Linux
>>> jvm_prop_os_version 3.5.0-44-generic
>>> jvm_prop_path_separator :
>>> jvm_prop_sun_arch_data_model 64
>>> jvm_prop_sun_boot_class_path
>>>
>>>
>> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
>>> jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
>>> jvm_prop_sun_cpu_endian little
>>> jvm_prop_sun_cpu_isalist
>>> jvm_prop_sun_io_unicode_encoding UnicodeLittle
>>> jvm_prop_sun_java_command
>>> com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
>>> -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
>>> -zk_endpoints=192.168.0.13:2181
>>> -mesos_master_address=zk://192.168.0.13:2181/mesos/master
>>> -serverset_path=/aurora/scheduler
>>> -native_log_zk_group_path=/aurora/replicated-log
>>> -native_log_file_path=/usr/local/aurora-scheduler/db
>>> -backup_dir=/usr/local/aurora-scheduler/backups
>>> -thermos_executor_path=/usr/local/bin/thermos_executor
>>> -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
>>> jvm_prop_sun_java_launcher SUN_STANDARD
>>> jvm_prop_sun_jnu_encoding UTF-8
>>> jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
>>> jvm_prop_sun_os_patch_level unknown
>>> jvm_prop_user_country US
>>> jvm_prop_user_dir /home/lychee
>>> jvm_prop_user_home /root
>>> jvm_prop_user_language en
>>> jvm_prop_user_name root
>>> jvm_prop_user_timezone Etc/UTC
>>> jvm_threads_active 21
>>> jvm_threads_daemon 12
>>> jvm_threads_peak 25
>>> jvm_threads_started 867
>>> jvm_time_ms 1387559947214
>>> jvm_uptime_secs 38781
>>> log_compressed_entry_bytes_saved 2537635
>>> logging_dir /root
>>> mem_storage_consistent_read_operation_events 619
>>> mem_storage_consistent_read_operation_events_per_sec 0.0
>>> mem_storage_consistent_read_operation_nanos_per_event 0.0
>>> mem_storage_consistent_read_operation_nanos_total 272565131
>>> mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
>>> mem_storage_delete_all_tasks_events 1
>>> mem_storage_delete_all_tasks_events_per_sec 0.0
>>> mem_storage_delete_all_tasks_nanos_per_event 0.0
>>> mem_storage_delete_all_tasks_nanos_total 2323293
>>> mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
>>> mem_storage_delete_tasks_events 179
>>> mem_storage_delete_tasks_events_per_sec 0.0
>>> mem_storage_delete_tasks_nanos_per_event 0.0
>>> mem_storage_delete_tasks_nanos_total 8638495
>>> mem_storage_delete_tasks_nanos_total_per_sec 0.0
>>> mem_storage_fetch_tasks_events 3219
>>> mem_storage_fetch_tasks_events_per_sec 0.0
>>> mem_storage_fetch_tasks_nanos_per_event 0.0
>>> mem_storage_fetch_tasks_nanos_total 193613049
>>> mem_storage_fetch_tasks_nanos_total_per_sec 0.0
>>> mem_storage_mutate_tasks_events 677
>>> mem_storage_mutate_tasks_events_per_sec 0.0
>>> mem_storage_mutate_tasks_nanos_per_event 0.0
>>> mem_storage_mutate_tasks_nanos_total 149943165
>>> mem_storage_mutate_tasks_nanos_total_per_sec 0.0
>>> mem_storage_save_tasks_events 306
>>> mem_storage_save_tasks_events_per_sec 0.0
>>> mem_storage_save_tasks_nanos_per_event 0.0
>>> mem_storage_save_tasks_nanos_total 165988968
>>> mem_storage_save_tasks_nanos_total_per_sec 0.0
>>> mem_storage_weakly_consistent_read_operation_events 1538
>>> mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
>>> mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
>>> mem_storage_weakly_consistent_read_operation_nanos_total 138584844
>>> mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
>>> mem_storage_write_operation_events 2932
>>> mem_storage_write_operation_events_per_sec 0.0
>>> mem_storage_write_operation_nanos_per_event 0.0
>>> mem_storage_write_operation_nanos_total 28077154879
>>> mem_storage_write_operation_nanos_total_per_sec 0.0
>>> new_instance_id_set 0
>>> offer_accept_races 0
>>> old_instance_id_set 0
>>> outstanding_offers 1
>>> preemptor_attempts 0
>>> preemptor_failed_preemptions 0
>>> preemptor_no_slots_found 0
>>> preemptor_tasks_preempted 0
>>> process_cpu_cores_utilized 0.0
>>> process_cpu_time_nanos 73230000000
>>> process_max_fd_count 4096
>>> process_open_fd_count 157
>>> read_lock_wait_nanos 545041
>>> resources_allocated_quota_cpu 0
>>> resources_allocated_quota_disk_gb 0
>>> resources_allocated_quota_ram_gb 0
>>> resources_dedicated_consumed_cpu 0
>>> resources_dedicated_consumed_disk_gb 0
>>> resources_dedicated_consumed_ram_gb 0
>>> resources_free_pool_consumed_cpu 1
>>> resources_free_pool_consumed_disk_gb 0
>>> resources_free_pool_consumed_ram_gb 0
>>> resources_quota_consumed_cpu 0
>>> resources_quota_consumed_disk_gb 0
>>> resources_quota_consumed_ram_gb 0
>>> resources_total_consumed_cpu 1
>>> resources_total_consumed_disk_gb 0
>>> resources_total_consumed_ram_gb 0
>>> schedule_attempts_failed 0
>>> schedule_attempts_fired 135
>>> schedule_queue_size 1
>>> scheduler_backup_failed 0
>>> scheduler_backup_success 10
>>> scheduler_driver_kill_failures 0
>>> scheduler_failed_offers 0
>>> scheduler_framework_disconnects 0
>>> scheduler_framework_reregisters 0
>>> scheduler_illegal_task_state_transitions 200
>>> scheduler_log_bad_frames_read 0
>>> scheduler_log_bytes_read 292813
>>> scheduler_log_bytes_written 1080061
>>> scheduler_log_deflated_entries_read 1
>>> scheduler_log_entries_read 175
>>> scheduler_log_entries_written 555
>>> scheduler_log_native_append_events 556
>>> scheduler_log_native_append_events_per_sec 0.0
>>> scheduler_log_native_append_failures 0
>>> scheduler_log_native_append_nanos_per_event 0.0
>>> scheduler_log_native_append_nanos_total 25082628348
>>> scheduler_log_native_append_nanos_total_per_sec 0.0
>>> scheduler_log_native_append_timeouts 0
>>> scheduler_log_native_native_entries_skipped 1
>>> scheduler_log_native_read_events 176
>>> scheduler_log_native_read_events_per_sec 0.0
>>> scheduler_log_native_read_failures 0
>>> scheduler_log_native_read_nanos_per_event 0.0
>>> scheduler_log_native_read_nanos_total 73255842
>>> scheduler_log_native_read_nanos_total_per_sec 0.0
>>> scheduler_log_native_read_timeouts 0
>>> scheduler_log_native_truncate_events 10
>>> scheduler_log_native_truncate_events_per_sec 0.0
>>> scheduler_log_native_truncate_failures 0
>>> scheduler_log_native_truncate_nanos_per_event 0.0
>>> scheduler_log_native_truncate_nanos_total 276600750
>>> scheduler_log_native_truncate_nanos_total_per_sec 0.0
>>> scheduler_log_native_truncate_timeouts 0
>>> scheduler_log_recover_events 1
>>> scheduler_log_recover_events_per_sec 0.0
>>> scheduler_log_recover_nanos_per_event 0.0
>>> scheduler_log_recover_nanos_total 565333612
>>> scheduler_log_recover_nanos_total_per_sec 0.0
>>> scheduler_log_save_framework_id_events 2
>>> scheduler_log_save_framework_id_events_per_sec 0.0
>>> scheduler_log_save_framework_id_nanos_per_event 0.0
>>> scheduler_log_save_framework_id_nanos_total 437292
>>> scheduler_log_save_framework_id_nanos_total_per_sec 0.0
>>> scheduler_log_snapshot_events 10
>>> scheduler_log_snapshot_events_per_sec 0.0
>>> scheduler_log_snapshot_nanos_per_event 0.0
>>> scheduler_log_snapshot_nanos_total 961465418
>>> scheduler_log_snapshot_nanos_total_per_sec 0.0
>>> scheduler_log_snapshot_persist_events 10
>>> scheduler_log_snapshot_persist_events_per_sec 0.0
>>> scheduler_log_snapshot_persist_nanos_per_event 0.0
>>> scheduler_log_snapshot_persist_nanos_total 840444765
>>> scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
>>> scheduler_log_snapshots 10
>>> scheduler_log_tasks_mutate_events 677
>>> scheduler_log_tasks_mutate_events_per_sec 0.0
>>> scheduler_log_tasks_mutate_nanos_per_event 0.0
>>> scheduler_log_tasks_mutate_nanos_total 182346727
>>> scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
>>> scheduler_log_tasks_remove_events 179
>>> scheduler_log_tasks_remove_events_per_sec 0.0
>>> scheduler_log_tasks_remove_nanos_per_event 0.0
>>> scheduler_log_tasks_remove_nanos_total 13588163
>>> scheduler_log_tasks_remove_nanos_total_per_sec 0.0
>>> scheduler_log_tasks_save_events 305
>>> scheduler_log_tasks_save_events_per_sec 0.0
>>> scheduler_log_tasks_save_nanos_per_event 0.0
>>> scheduler_log_tasks_save_nanos_total 114479852
>>> scheduler_log_tasks_save_nanos_total_per_sec 0.0
>>> scheduler_log_un_snapshotted_transactions 36
>>> scheduler_lost_executors 0
>>> scheduler_max_ASSIGNED_waiting_ms 0
>>> scheduler_max_KILLING_waiting_ms 0
>>> scheduler_max_PREEMPTING_waiting_ms 0
>>> scheduler_max_RESTARTING_waiting_ms 0
>>> scheduler_resource_offers 397
>>> scheduler_resource_offers_events 397
>>> scheduler_resource_offers_events_per_sec 0.0
>>> scheduler_resource_offers_nanos_per_event 0.0
>>> scheduler_resource_offers_nanos_total 305282357
>>> scheduler_resource_offers_nanos_total_per_sec 0.0
>>> scheduler_save_host_attribute_events 397
>>> scheduler_save_host_attribute_events_per_sec 0.0
>>> scheduler_save_host_attribute_nanos_per_event 0.0
>>> scheduler_save_host_attribute_nanos_total 12466197
>>> scheduler_save_host_attribute_nanos_total_per_sec 0.0
>>> scheduler_status_update_events 426
>>> scheduler_status_update_events_per_sec 0.0
>>> scheduler_status_update_nanos_per_event 0.0
>>> scheduler_status_update_nanos_total 19334004116
>>> scheduler_status_update_nanos_total_per_sec 0.0
>>> scheduler_status_updates 0
>>> scheduler_thrift_createJob_events 2
>>> scheduler_thrift_createJob_events_per_sec 0.0
>>> scheduler_thrift_createJob_nanos_per_event 0.0
>>> scheduler_thrift_createJob_nanos_total 54102009
>>> scheduler_thrift_createJob_nanos_total_per_sec 0.0
>>> scheduler_thrift_getTasksStatus_events 2
>>> scheduler_thrift_getTasksStatus_events_per_sec 0.0
>>> scheduler_thrift_getTasksStatus_nanos_per_event 0.0
>>> scheduler_thrift_getTasksStatus_nanos_total 5684762
>>> scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
>>> scheduler_thrift_getVersion_events 4
>>> scheduler_thrift_getVersion_events_per_sec 0.0
>>> scheduler_thrift_getVersion_nanos_per_event 0.0
>>> scheduler_thrift_getVersion_nanos_total 13854714
>>> scheduler_thrift_getVersion_nanos_total_per_sec 0.0
>>> scheduler_thrift_killTasks_events 2
>>> scheduler_thrift_killTasks_events_per_sec 0.0
>>> scheduler_thrift_killTasks_nanos_per_event 0.0
>>> scheduler_thrift_killTasks_nanos_total 186407312
>>> scheduler_thrift_killTasks_nanos_total_per_sec 0.0
>>> shard_sanity_check_failures 0
>>> snapshot_apply_events 1
>>> snapshot_apply_events_per_sec 0.0
>>> snapshot_apply_nanos_per_event 0.0
>>> snapshot_apply_nanos_total 93051831
>>> snapshot_apply_nanos_total_per_sec 0.0
>>> snapshot_create_events 10
>>> snapshot_create_events_per_sec 0.0
>>> snapshot_create_nanos_per_event 0.0
>>> snapshot_create_nanos_total 37066472
>>> snapshot_create_nanos_total_per_sec 0.0
>>> system_env_GLOG_v 0
>>> system_env_HOME /home/lychee
>>> system_env_JAVA_OPTS -server -Xmx1g -Xms1g
>>> -Djava.library.path=/usr/local/lib
>>> system_env_LANG en_US.UTF-8
>>> system_env_LIBPROCESS_IP 192.168.0.18
>>> system_env_LIBPROCESS_PORT 8083
>>> system_env_LOGNAME root
>>> system_env_MAIL /var/mail/root
>>> system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
>>> system_env_OLDPWD /usr/local/aurora-scheduler
>>> system_env_PATH
>>> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>>> system_env_PWD /home/lychee
>>> system_env_SHELL /bin/bash
>>> system_env_SHLVL 0
>>> system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
>>> system_env_SUDO_GID 1002
>>> system_env_SUDO_UID 1002
>>> system_env_SUDO_USER lychee
>>> system_env_TERM xterm
>>> system_env_USER root
>>> system_env_USERNAME root
>>> system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
>>> system_free_physical_memory_mb 148
>>> system_free_swap_mb 8063
>>> system_load_avg 2.22
>>> task_queries_all 45
>>> task_queries_by_id 3668
>>> task_queries_by_job 183
>>> task_schedule_attempt_events 135
>>> task_schedule_attempt_events_per_sec 0.0
>>> task_schedule_attempt_nanos_per_event 0.0
>>> task_schedule_attempt_nanos_total 6289005821
>>> task_schedule_attempt_nanos_total_per_sec 0.0
>>> task_store_ASSIGNED 0
>>> task_store_FAILED 0
>>> task_store_FINISHED 200
>>> task_store_INIT 0
>>> task_store_KILLED 0
>>> task_store_KILLING 0
>>> task_store_LOST 0
>>> task_store_PENDING 1
>>> task_store_PREEMPTING 0
>>> task_store_RESTARTING 0
>>> task_store_RUNNING 0
>>> task_store_STARTING 0
>>> task_store_THROTTLED 0
>>> task_store_UNKNOWN 0
>>> timed_out_tasks 0
>>> timeout_queue_size 338
>>> transient_states 0
>>> uncaught_exceptions 0
>>> variable_scrape_events 38778
>>> variable_scrape_events_per_sec 1.0000103021061324
>>> variable_scrape_micros_per_event 455.0001369564521
>>> variable_scrape_micros_total 18809481
>>> variable_scrape_micros_total_per_sec 455.00482441615327
>>> work_queue_depth 0
>>> write_lock_wait_nanos 13138744
>>>
>>>
>>>
>>>
>>> On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
>>>> wrote:
>>>
>>>> Hi Yingbo,
>>>>
>>>> Do you mind sharing your .aurora file? Perhaps your are requesting
>>>> Resources or Constraints that are infeasible for your local setup to
>>>> satisfy?
>>>>
>>>> Also, you might want to check that your scheduler is receiving offers
>>> from
>>>> mesos master. You could check your scheduler log file to see if any
>>> offers
>>>> are coming. Alternatively, you might want to look at
>>>> "scheduler_resource_offers" stat exposed by the scheduler /vars
>> endpoint
>>>> (http://<your_local_scheduler_ip:port>/vars).
>>>>
>>>> Thanks,
>>>> Maxim
>>>>
>>>>
>>>>
>>>> On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
>>>>
>>>>> hi,
>>>>> Hope you guys have a fun night at city hall.
>>>>> While you might be drunk, can anyone help me a little bit on
>> setting
>>>> up
>>>>> the local dev environment to debug aurora ?
>>>>> My current set up.
>>>>> Machine A: 64bit ubuntu server running mesos-master,
>>>>> Machine B: 64bit ubuntu server running mesos-slave, thermo
>>>>> observer server;
>>>>> Machine C: 32bit ubuntu server running zookeeper.
>>>>> Machine D: Mac air for developing aurora.
>>>>>
>>>>> I tried to run aurora-scheduler on Machine A and also created a
>>>> hello
>>>>> job from machine A. the job was submitted successfully. however the
>> job
>>>>> keeps staying at "PENDING" status. i checked thermo observer from
>> port
>>>>> 1336, it says "No matching hosts".
>>>>>
>>>>> I spent a couple of hours and still can't figure out the
>> reason
>>> so
>>>>> I decided to settup the dev environment on my Mac so that I can trace
>>> the
>>>>> error in IntelliJ. Can someone give me some hint on that? I am not
>>> quite
>>>>> familiar with IntelliJ and java.
>>>>>
>>>>>
>>>>>
>>>>> best
>>>>> Yingbo
>>>>
>>>>
>>>
>>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Yingbo Wang <yb...@gmail.com>.
It is a good catch! Thanks Brian.
The task indeed restarts every 5 minutes. I guess the job just doesn't
can't find the running service that's why it says "Not matching hosts"?
I will try it to see if it works.
On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman <wi...@twopensource.com>wrote:
> There is an executor_id in the state there, so it looks like it is managing
> to launch the tasks. Since you've listed your job as a Service(), it means
> that every task will be rescheduled after it exits, and since it looks like
> your task just immediately returns, it's probably launching dozens of them
> per minute if not more. You can change this behavior by changing Service
> to Job. (Service is just an alias for Job(service=True).)
>
> ~brian
>
>
>
> On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
>
> > Thanks Maxim.
> >
> > did you mean hello.aurora?
> >
> >
> > hello = Process(
> > name = 'hello',
> > cmdline = """
> > echo "ls file " ;
> > whoami;
> > hostname;
> > """)
> >
> > task = SequentialTask(
> > processes = [hello],
> > resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
> >
> > jobs = [Service(
> > task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
> > name = 'hello')]
> >
> > The slave machine has enough resource to satisfy this request.
> >
> > I did encounter an issue saying "Constraint Not Satisfied, host". After I
> > change my hostname of both master and slave machines, this problem seems
> to
> > be solved, and now the problem becomes "No matching hosts"
> >
> >
> >
> > Here's the offers that scheduler received:
> >
> > [
> >
> > -
> > {
> > - id: "201312200632-302033088-5050-15916-396",
> > - framework_id: "201312110654-302033088-5050-31688-0000",
> > - slave_id: "201312170824-302033088-5050-25318-0",
> > - hostname: "192.168.0.11",
> > - resources:
> > [
> > -
> > {
> > - name: "cpus",
> > - scalar: 7.75
> > },
> > -
> > {
> > - name: "disk",
> > - scalar: 917136
> > },
> > -
> > {
> > - name: "mem",
> > - scalar: 14814
> > },
> > -
> > {
> > - name: "ports",
> > - ranges:
> > [
> > - "31000-32000"
> > ]
> > }
> > ],
> > - attributes:
> > [
> > -
> > {
> > - name: "host",
> > - text: "192.168.0.11"
> > },
> > -
> > {
> > - name: "rack",
> > - text: "a"
> > }
> > ],
> > - executor_ids:
> > [
> > -
> >
> >
> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
> > ]
> > }
> >
> > ]
> >
> >
> > and the /vars are:
> >
> >
> > async_tasks_completed 393
> > both_instance_ids_set 0
> > build_date null
> > build_git_branchname null
> > build_git_revision null
> > build_git_revision_number null
> > build_git_tag null
> > build_machine null
> > build_path null
> > build_time null
> > build_timestamp null
> > build_user_name null
> > cron_job_launch_failures 0
> > cron_jobs_triggered 0
> > cron_num_pending_runs 0
> > empty_slots_large 0
> > empty_slots_medium 1
> > empty_slots_small 7
> > empty_slots_xlarge 0
> > framework_registered 1
> > http_200_responses_events 22
> > http_200_responses_events_per_sec 0.0
> > http_200_responses_nanos_per_event 0.0
> > http_200_responses_nanos_total 981542679
> > http_200_responses_nanos_total_per_sec 0.0
> > instance_ids_inconsistent 0
> > jvm_available_processors 4
> > jvm_class_loaded_count 6658
> > jvm_class_total_loaded_count 6658
> > jvm_class_unloaded_count 0
> > jvm_gc_PS_MarkSweep_collection_count 0
> > jvm_gc_PS_MarkSweep_collection_time_ms 0
> > jvm_gc_PS_Scavenge_collection_count 7
> > jvm_gc_PS_Scavenge_collection_time_ms 534
> > jvm_gc_collection_count 7
> > jvm_gc_collection_time_ms 534
> > jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
> > jvm_memory_free_mb 688
> > jvm_memory_heap_mb_committed 968
> > jvm_memory_heap_mb_max 968
> > jvm_memory_heap_mb_used 279
> > jvm_memory_max_mb 968
> > jvm_memory_mb_total 968
> > jvm_memory_non_heap_mb_committed 39
> > jvm_memory_non_heap_mb_max 130
> > jvm_memory_non_heap_mb_used 38
> > jvm_prop_awt_toolkit sun.awt.X11.XToolkit
> > jvm_prop_file_encoding UTF-8
> > jvm_prop_file_encoding_pkg sun.io
> > jvm_prop_file_separator /
> > jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
> > jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
> > jvm_prop_java_class_path
> >
> >
> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
> > jvm_prop_java_class_version 51.0
> > jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
> > jvm_prop_java_ext_dirs
> > /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
> > jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
> > jvm_prop_java_io_tmpdir /tmp
> > jvm_prop_java_library_path /usr/local/lib
> > jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
> > jvm_prop_java_runtime_version 1.7.0_45-b18
> > jvm_prop_java_specification_name Java Platform API Specification
> > jvm_prop_java_specification_vendor Oracle Corporation
> > jvm_prop_java_specification_version 1.7
> > jvm_prop_java_vendor Oracle Corporation
> > jvm_prop_java_vendor_url http://java.oracle.com/
> > jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
> > jvm_prop_java_version 1.7.0_45
> > jvm_prop_java_vm_info mixed mode
> > jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
> > jvm_prop_java_vm_specification_name Java Virtual Machine Specification
> > jvm_prop_java_vm_specification_vendor Oracle Corporation
> > jvm_prop_java_vm_specification_version 1.7
> > jvm_prop_java_vm_vendor Oracle Corporation
> > jvm_prop_java_vm_version 24.45-b08
> > jvm_prop_line_separator
> >
> > jvm_prop_os_arch amd64
> > jvm_prop_os_name Linux
> > jvm_prop_os_version 3.5.0-44-generic
> > jvm_prop_path_separator :
> > jvm_prop_sun_arch_data_model 64
> > jvm_prop_sun_boot_class_path
> >
> >
> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
> > jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
> > jvm_prop_sun_cpu_endian little
> > jvm_prop_sun_cpu_isalist
> > jvm_prop_sun_io_unicode_encoding UnicodeLittle
> > jvm_prop_sun_java_command
> > com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
> > -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
> > -zk_endpoints=192.168.0.13:2181
> > -mesos_master_address=zk://192.168.0.13:2181/mesos/master
> > -serverset_path=/aurora/scheduler
> > -native_log_zk_group_path=/aurora/replicated-log
> > -native_log_file_path=/usr/local/aurora-scheduler/db
> > -backup_dir=/usr/local/aurora-scheduler/backups
> > -thermos_executor_path=/usr/local/bin/thermos_executor
> > -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
> > jvm_prop_sun_java_launcher SUN_STANDARD
> > jvm_prop_sun_jnu_encoding UTF-8
> > jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
> > jvm_prop_sun_os_patch_level unknown
> > jvm_prop_user_country US
> > jvm_prop_user_dir /home/lychee
> > jvm_prop_user_home /root
> > jvm_prop_user_language en
> > jvm_prop_user_name root
> > jvm_prop_user_timezone Etc/UTC
> > jvm_threads_active 21
> > jvm_threads_daemon 12
> > jvm_threads_peak 25
> > jvm_threads_started 867
> > jvm_time_ms 1387559947214
> > jvm_uptime_secs 38781
> > log_compressed_entry_bytes_saved 2537635
> > logging_dir /root
> > mem_storage_consistent_read_operation_events 619
> > mem_storage_consistent_read_operation_events_per_sec 0.0
> > mem_storage_consistent_read_operation_nanos_per_event 0.0
> > mem_storage_consistent_read_operation_nanos_total 272565131
> > mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
> > mem_storage_delete_all_tasks_events 1
> > mem_storage_delete_all_tasks_events_per_sec 0.0
> > mem_storage_delete_all_tasks_nanos_per_event 0.0
> > mem_storage_delete_all_tasks_nanos_total 2323293
> > mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
> > mem_storage_delete_tasks_events 179
> > mem_storage_delete_tasks_events_per_sec 0.0
> > mem_storage_delete_tasks_nanos_per_event 0.0
> > mem_storage_delete_tasks_nanos_total 8638495
> > mem_storage_delete_tasks_nanos_total_per_sec 0.0
> > mem_storage_fetch_tasks_events 3219
> > mem_storage_fetch_tasks_events_per_sec 0.0
> > mem_storage_fetch_tasks_nanos_per_event 0.0
> > mem_storage_fetch_tasks_nanos_total 193613049
> > mem_storage_fetch_tasks_nanos_total_per_sec 0.0
> > mem_storage_mutate_tasks_events 677
> > mem_storage_mutate_tasks_events_per_sec 0.0
> > mem_storage_mutate_tasks_nanos_per_event 0.0
> > mem_storage_mutate_tasks_nanos_total 149943165
> > mem_storage_mutate_tasks_nanos_total_per_sec 0.0
> > mem_storage_save_tasks_events 306
> > mem_storage_save_tasks_events_per_sec 0.0
> > mem_storage_save_tasks_nanos_per_event 0.0
> > mem_storage_save_tasks_nanos_total 165988968
> > mem_storage_save_tasks_nanos_total_per_sec 0.0
> > mem_storage_weakly_consistent_read_operation_events 1538
> > mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
> > mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
> > mem_storage_weakly_consistent_read_operation_nanos_total 138584844
> > mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
> > mem_storage_write_operation_events 2932
> > mem_storage_write_operation_events_per_sec 0.0
> > mem_storage_write_operation_nanos_per_event 0.0
> > mem_storage_write_operation_nanos_total 28077154879
> > mem_storage_write_operation_nanos_total_per_sec 0.0
> > new_instance_id_set 0
> > offer_accept_races 0
> > old_instance_id_set 0
> > outstanding_offers 1
> > preemptor_attempts 0
> > preemptor_failed_preemptions 0
> > preemptor_no_slots_found 0
> > preemptor_tasks_preempted 0
> > process_cpu_cores_utilized 0.0
> > process_cpu_time_nanos 73230000000
> > process_max_fd_count 4096
> > process_open_fd_count 157
> > read_lock_wait_nanos 545041
> > resources_allocated_quota_cpu 0
> > resources_allocated_quota_disk_gb 0
> > resources_allocated_quota_ram_gb 0
> > resources_dedicated_consumed_cpu 0
> > resources_dedicated_consumed_disk_gb 0
> > resources_dedicated_consumed_ram_gb 0
> > resources_free_pool_consumed_cpu 1
> > resources_free_pool_consumed_disk_gb 0
> > resources_free_pool_consumed_ram_gb 0
> > resources_quota_consumed_cpu 0
> > resources_quota_consumed_disk_gb 0
> > resources_quota_consumed_ram_gb 0
> > resources_total_consumed_cpu 1
> > resources_total_consumed_disk_gb 0
> > resources_total_consumed_ram_gb 0
> > schedule_attempts_failed 0
> > schedule_attempts_fired 135
> > schedule_queue_size 1
> > scheduler_backup_failed 0
> > scheduler_backup_success 10
> > scheduler_driver_kill_failures 0
> > scheduler_failed_offers 0
> > scheduler_framework_disconnects 0
> > scheduler_framework_reregisters 0
> > scheduler_illegal_task_state_transitions 200
> > scheduler_log_bad_frames_read 0
> > scheduler_log_bytes_read 292813
> > scheduler_log_bytes_written 1080061
> > scheduler_log_deflated_entries_read 1
> > scheduler_log_entries_read 175
> > scheduler_log_entries_written 555
> > scheduler_log_native_append_events 556
> > scheduler_log_native_append_events_per_sec 0.0
> > scheduler_log_native_append_failures 0
> > scheduler_log_native_append_nanos_per_event 0.0
> > scheduler_log_native_append_nanos_total 25082628348
> > scheduler_log_native_append_nanos_total_per_sec 0.0
> > scheduler_log_native_append_timeouts 0
> > scheduler_log_native_native_entries_skipped 1
> > scheduler_log_native_read_events 176
> > scheduler_log_native_read_events_per_sec 0.0
> > scheduler_log_native_read_failures 0
> > scheduler_log_native_read_nanos_per_event 0.0
> > scheduler_log_native_read_nanos_total 73255842
> > scheduler_log_native_read_nanos_total_per_sec 0.0
> > scheduler_log_native_read_timeouts 0
> > scheduler_log_native_truncate_events 10
> > scheduler_log_native_truncate_events_per_sec 0.0
> > scheduler_log_native_truncate_failures 0
> > scheduler_log_native_truncate_nanos_per_event 0.0
> > scheduler_log_native_truncate_nanos_total 276600750
> > scheduler_log_native_truncate_nanos_total_per_sec 0.0
> > scheduler_log_native_truncate_timeouts 0
> > scheduler_log_recover_events 1
> > scheduler_log_recover_events_per_sec 0.0
> > scheduler_log_recover_nanos_per_event 0.0
> > scheduler_log_recover_nanos_total 565333612
> > scheduler_log_recover_nanos_total_per_sec 0.0
> > scheduler_log_save_framework_id_events 2
> > scheduler_log_save_framework_id_events_per_sec 0.0
> > scheduler_log_save_framework_id_nanos_per_event 0.0
> > scheduler_log_save_framework_id_nanos_total 437292
> > scheduler_log_save_framework_id_nanos_total_per_sec 0.0
> > scheduler_log_snapshot_events 10
> > scheduler_log_snapshot_events_per_sec 0.0
> > scheduler_log_snapshot_nanos_per_event 0.0
> > scheduler_log_snapshot_nanos_total 961465418
> > scheduler_log_snapshot_nanos_total_per_sec 0.0
> > scheduler_log_snapshot_persist_events 10
> > scheduler_log_snapshot_persist_events_per_sec 0.0
> > scheduler_log_snapshot_persist_nanos_per_event 0.0
> > scheduler_log_snapshot_persist_nanos_total 840444765
> > scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
> > scheduler_log_snapshots 10
> > scheduler_log_tasks_mutate_events 677
> > scheduler_log_tasks_mutate_events_per_sec 0.0
> > scheduler_log_tasks_mutate_nanos_per_event 0.0
> > scheduler_log_tasks_mutate_nanos_total 182346727
> > scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
> > scheduler_log_tasks_remove_events 179
> > scheduler_log_tasks_remove_events_per_sec 0.0
> > scheduler_log_tasks_remove_nanos_per_event 0.0
> > scheduler_log_tasks_remove_nanos_total 13588163
> > scheduler_log_tasks_remove_nanos_total_per_sec 0.0
> > scheduler_log_tasks_save_events 305
> > scheduler_log_tasks_save_events_per_sec 0.0
> > scheduler_log_tasks_save_nanos_per_event 0.0
> > scheduler_log_tasks_save_nanos_total 114479852
> > scheduler_log_tasks_save_nanos_total_per_sec 0.0
> > scheduler_log_un_snapshotted_transactions 36
> > scheduler_lost_executors 0
> > scheduler_max_ASSIGNED_waiting_ms 0
> > scheduler_max_KILLING_waiting_ms 0
> > scheduler_max_PREEMPTING_waiting_ms 0
> > scheduler_max_RESTARTING_waiting_ms 0
> > scheduler_resource_offers 397
> > scheduler_resource_offers_events 397
> > scheduler_resource_offers_events_per_sec 0.0
> > scheduler_resource_offers_nanos_per_event 0.0
> > scheduler_resource_offers_nanos_total 305282357
> > scheduler_resource_offers_nanos_total_per_sec 0.0
> > scheduler_save_host_attribute_events 397
> > scheduler_save_host_attribute_events_per_sec 0.0
> > scheduler_save_host_attribute_nanos_per_event 0.0
> > scheduler_save_host_attribute_nanos_total 12466197
> > scheduler_save_host_attribute_nanos_total_per_sec 0.0
> > scheduler_status_update_events 426
> > scheduler_status_update_events_per_sec 0.0
> > scheduler_status_update_nanos_per_event 0.0
> > scheduler_status_update_nanos_total 19334004116
> > scheduler_status_update_nanos_total_per_sec 0.0
> > scheduler_status_updates 0
> > scheduler_thrift_createJob_events 2
> > scheduler_thrift_createJob_events_per_sec 0.0
> > scheduler_thrift_createJob_nanos_per_event 0.0
> > scheduler_thrift_createJob_nanos_total 54102009
> > scheduler_thrift_createJob_nanos_total_per_sec 0.0
> > scheduler_thrift_getTasksStatus_events 2
> > scheduler_thrift_getTasksStatus_events_per_sec 0.0
> > scheduler_thrift_getTasksStatus_nanos_per_event 0.0
> > scheduler_thrift_getTasksStatus_nanos_total 5684762
> > scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
> > scheduler_thrift_getVersion_events 4
> > scheduler_thrift_getVersion_events_per_sec 0.0
> > scheduler_thrift_getVersion_nanos_per_event 0.0
> > scheduler_thrift_getVersion_nanos_total 13854714
> > scheduler_thrift_getVersion_nanos_total_per_sec 0.0
> > scheduler_thrift_killTasks_events 2
> > scheduler_thrift_killTasks_events_per_sec 0.0
> > scheduler_thrift_killTasks_nanos_per_event 0.0
> > scheduler_thrift_killTasks_nanos_total 186407312
> > scheduler_thrift_killTasks_nanos_total_per_sec 0.0
> > shard_sanity_check_failures 0
> > snapshot_apply_events 1
> > snapshot_apply_events_per_sec 0.0
> > snapshot_apply_nanos_per_event 0.0
> > snapshot_apply_nanos_total 93051831
> > snapshot_apply_nanos_total_per_sec 0.0
> > snapshot_create_events 10
> > snapshot_create_events_per_sec 0.0
> > snapshot_create_nanos_per_event 0.0
> > snapshot_create_nanos_total 37066472
> > snapshot_create_nanos_total_per_sec 0.0
> > system_env_GLOG_v 0
> > system_env_HOME /home/lychee
> > system_env_JAVA_OPTS -server -Xmx1g -Xms1g
> > -Djava.library.path=/usr/local/lib
> > system_env_LANG en_US.UTF-8
> > system_env_LIBPROCESS_IP 192.168.0.18
> > system_env_LIBPROCESS_PORT 8083
> > system_env_LOGNAME root
> > system_env_MAIL /var/mail/root
> > system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> > system_env_OLDPWD /usr/local/aurora-scheduler
> > system_env_PATH
> > /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > system_env_PWD /home/lychee
> > system_env_SHELL /bin/bash
> > system_env_SHLVL 0
> > system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
> > system_env_SUDO_GID 1002
> > system_env_SUDO_UID 1002
> > system_env_SUDO_USER lychee
> > system_env_TERM xterm
> > system_env_USER root
> > system_env_USERNAME root
> > system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
> > system_free_physical_memory_mb 148
> > system_free_swap_mb 8063
> > system_load_avg 2.22
> > task_queries_all 45
> > task_queries_by_id 3668
> > task_queries_by_job 183
> > task_schedule_attempt_events 135
> > task_schedule_attempt_events_per_sec 0.0
> > task_schedule_attempt_nanos_per_event 0.0
> > task_schedule_attempt_nanos_total 6289005821
> > task_schedule_attempt_nanos_total_per_sec 0.0
> > task_store_ASSIGNED 0
> > task_store_FAILED 0
> > task_store_FINISHED 200
> > task_store_INIT 0
> > task_store_KILLED 0
> > task_store_KILLING 0
> > task_store_LOST 0
> > task_store_PENDING 1
> > task_store_PREEMPTING 0
> > task_store_RESTARTING 0
> > task_store_RUNNING 0
> > task_store_STARTING 0
> > task_store_THROTTLED 0
> > task_store_UNKNOWN 0
> > timed_out_tasks 0
> > timeout_queue_size 338
> > transient_states 0
> > uncaught_exceptions 0
> > variable_scrape_events 38778
> > variable_scrape_events_per_sec 1.0000103021061324
> > variable_scrape_micros_per_event 455.0001369564521
> > variable_scrape_micros_total 18809481
> > variable_scrape_micros_total_per_sec 455.00482441615327
> > work_queue_depth 0
> > write_lock_wait_nanos 13138744
> >
> >
> >
> >
> > On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
> > >wrote:
> >
> > > Hi Yingbo,
> > >
> > > Do you mind sharing your .aurora file? Perhaps your are requesting
> > > Resources or Constraints that are infeasible for your local setup to
> > > satisfy?
> > >
> > > Also, you might want to check that your scheduler is receiving offers
> > from
> > > mesos master. You could check your scheduler log file to see if any
> > offers
> > > are coming. Alternatively, you might want to look at
> > > "scheduler_resource_offers" stat exposed by the scheduler /vars
> endpoint
> > > (http://<your_local_scheduler_ip:port>/vars).
> > >
> > > Thanks,
> > > Maxim
> > >
> > >
> > >
> > > On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> > >
> > > > hi,
> > > > Hope you guys have a fun night at city hall.
> > > > While you might be drunk, can anyone help me a little bit on
> setting
> > > up
> > > > the local dev environment to debug aurora ?
> > > > My current set up.
> > > > Machine A: 64bit ubuntu server running mesos-master,
> > > > Machine B: 64bit ubuntu server running mesos-slave, thermo
> > > > observer server;
> > > > Machine C: 32bit ubuntu server running zookeeper.
> > > > Machine D: Mac air for developing aurora.
> > > >
> > > > I tried to run aurora-scheduler on Machine A and also created a
> > > hello
> > > > job from machine A. the job was submitted successfully. however the
> job
> > > > keeps staying at "PENDING" status. i checked thermo observer from
> port
> > > > 1336, it says "No matching hosts".
> > > >
> > > > I spent a couple of hours and still can't figure out the
> reason
> > so
> > > > I decided to settup the dev environment on my Mac so that I can trace
> > the
> > > > error in IntelliJ. Can someone give me some hint on that? I am not
> > quite
> > > > familiar with IntelliJ and java.
> > > >
> > > >
> > > >
> > > > best
> > > > Yingbo
> > >
> > >
> >
>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Brian Wickman <wi...@twopensource.com>.
There is an executor_id in the state there, so it looks like it is managing
to launch the tasks. Since you've listed your job as a Service(), it means
that every task will be rescheduled after it exits, and since it looks like
your task just immediately returns, it's probably launching dozens of them
per minute if not more. You can change this behavior by changing Service
to Job. (Service is just an alias for Job(service=True).)
~brian
On Fri, Dec 20, 2013 at 9:20 AM, Yingbo Wang <yb...@gmail.com> wrote:
> Thanks Maxim.
>
> did you mean hello.aurora?
>
>
> hello = Process(
> name = 'hello',
> cmdline = """
> echo "ls file " ;
> whoami;
> hostname;
> """)
>
> task = SequentialTask(
> processes = [hello],
> resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
>
> jobs = [Service(
> task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
> name = 'hello')]
>
> The slave machine has enough resource to satisfy this request.
>
> I did encounter an issue saying "Constraint Not Satisfied, host". After I
> change my hostname of both master and slave machines, this problem seems to
> be solved, and now the problem becomes "No matching hosts"
>
>
>
> Here's the offers that scheduler received:
>
> [
>
> -
> {
> - id: "201312200632-302033088-5050-15916-396",
> - framework_id: "201312110654-302033088-5050-31688-0000",
> - slave_id: "201312170824-302033088-5050-25318-0",
> - hostname: "192.168.0.11",
> - resources:
> [
> -
> {
> - name: "cpus",
> - scalar: 7.75
> },
> -
> {
> - name: "disk",
> - scalar: 917136
> },
> -
> {
> - name: "mem",
> - scalar: 14814
> },
> -
> {
> - name: "ports",
> - ranges:
> [
> - "31000-32000"
> ]
> }
> ],
> - attributes:
> [
> -
> {
> - name: "host",
> - text: "192.168.0.11"
> },
> -
> {
> - name: "rack",
> - text: "a"
> }
> ],
> - executor_ids:
> [
> -
>
> "thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
> ]
> }
>
> ]
>
>
> and the /vars are:
>
>
> async_tasks_completed 393
> both_instance_ids_set 0
> build_date null
> build_git_branchname null
> build_git_revision null
> build_git_revision_number null
> build_git_tag null
> build_machine null
> build_path null
> build_time null
> build_timestamp null
> build_user_name null
> cron_job_launch_failures 0
> cron_jobs_triggered 0
> cron_num_pending_runs 0
> empty_slots_large 0
> empty_slots_medium 1
> empty_slots_small 7
> empty_slots_xlarge 0
> framework_registered 1
> http_200_responses_events 22
> http_200_responses_events_per_sec 0.0
> http_200_responses_nanos_per_event 0.0
> http_200_responses_nanos_total 981542679
> http_200_responses_nanos_total_per_sec 0.0
> instance_ids_inconsistent 0
> jvm_available_processors 4
> jvm_class_loaded_count 6658
> jvm_class_total_loaded_count 6658
> jvm_class_unloaded_count 0
> jvm_gc_PS_MarkSweep_collection_count 0
> jvm_gc_PS_MarkSweep_collection_time_ms 0
> jvm_gc_PS_Scavenge_collection_count 7
> jvm_gc_PS_Scavenge_collection_time_ms 534
> jvm_gc_collection_count 7
> jvm_gc_collection_time_ms 534
> jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
> jvm_memory_free_mb 688
> jvm_memory_heap_mb_committed 968
> jvm_memory_heap_mb_max 968
> jvm_memory_heap_mb_used 279
> jvm_memory_max_mb 968
> jvm_memory_mb_total 968
> jvm_memory_non_heap_mb_committed 39
> jvm_memory_non_heap_mb_max 130
> jvm_memory_non_heap_mb_used 38
> jvm_prop_awt_toolkit sun.awt.X11.XToolkit
> jvm_prop_file_encoding UTF-8
> jvm_prop_file_encoding_pkg sun.io
> jvm_prop_file_separator /
> jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
> jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
> jvm_prop_java_class_path
>
> /usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
> jvm_prop_java_class_version 51.0
> jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
> jvm_prop_java_ext_dirs
> /usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
> jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
> jvm_prop_java_io_tmpdir /tmp
> jvm_prop_java_library_path /usr/local/lib
> jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
> jvm_prop_java_runtime_version 1.7.0_45-b18
> jvm_prop_java_specification_name Java Platform API Specification
> jvm_prop_java_specification_vendor Oracle Corporation
> jvm_prop_java_specification_version 1.7
> jvm_prop_java_vendor Oracle Corporation
> jvm_prop_java_vendor_url http://java.oracle.com/
> jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
> jvm_prop_java_version 1.7.0_45
> jvm_prop_java_vm_info mixed mode
> jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
> jvm_prop_java_vm_specification_name Java Virtual Machine Specification
> jvm_prop_java_vm_specification_vendor Oracle Corporation
> jvm_prop_java_vm_specification_version 1.7
> jvm_prop_java_vm_vendor Oracle Corporation
> jvm_prop_java_vm_version 24.45-b08
> jvm_prop_line_separator
>
> jvm_prop_os_arch amd64
> jvm_prop_os_name Linux
> jvm_prop_os_version 3.5.0-44-generic
> jvm_prop_path_separator :
> jvm_prop_sun_arch_data_model 64
> jvm_prop_sun_boot_class_path
>
> /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
> jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
> jvm_prop_sun_cpu_endian little
> jvm_prop_sun_cpu_isalist
> jvm_prop_sun_io_unicode_encoding UnicodeLittle
> jvm_prop_sun_java_command
> com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
> -http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
> -zk_endpoints=192.168.0.13:2181
> -mesos_master_address=zk://192.168.0.13:2181/mesos/master
> -serverset_path=/aurora/scheduler
> -native_log_zk_group_path=/aurora/replicated-log
> -native_log_file_path=/usr/local/aurora-scheduler/db
> -backup_dir=/usr/local/aurora-scheduler/backups
> -thermos_executor_path=/usr/local/bin/thermos_executor
> -gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
> jvm_prop_sun_java_launcher SUN_STANDARD
> jvm_prop_sun_jnu_encoding UTF-8
> jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
> jvm_prop_sun_os_patch_level unknown
> jvm_prop_user_country US
> jvm_prop_user_dir /home/lychee
> jvm_prop_user_home /root
> jvm_prop_user_language en
> jvm_prop_user_name root
> jvm_prop_user_timezone Etc/UTC
> jvm_threads_active 21
> jvm_threads_daemon 12
> jvm_threads_peak 25
> jvm_threads_started 867
> jvm_time_ms 1387559947214
> jvm_uptime_secs 38781
> log_compressed_entry_bytes_saved 2537635
> logging_dir /root
> mem_storage_consistent_read_operation_events 619
> mem_storage_consistent_read_operation_events_per_sec 0.0
> mem_storage_consistent_read_operation_nanos_per_event 0.0
> mem_storage_consistent_read_operation_nanos_total 272565131
> mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
> mem_storage_delete_all_tasks_events 1
> mem_storage_delete_all_tasks_events_per_sec 0.0
> mem_storage_delete_all_tasks_nanos_per_event 0.0
> mem_storage_delete_all_tasks_nanos_total 2323293
> mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
> mem_storage_delete_tasks_events 179
> mem_storage_delete_tasks_events_per_sec 0.0
> mem_storage_delete_tasks_nanos_per_event 0.0
> mem_storage_delete_tasks_nanos_total 8638495
> mem_storage_delete_tasks_nanos_total_per_sec 0.0
> mem_storage_fetch_tasks_events 3219
> mem_storage_fetch_tasks_events_per_sec 0.0
> mem_storage_fetch_tasks_nanos_per_event 0.0
> mem_storage_fetch_tasks_nanos_total 193613049
> mem_storage_fetch_tasks_nanos_total_per_sec 0.0
> mem_storage_mutate_tasks_events 677
> mem_storage_mutate_tasks_events_per_sec 0.0
> mem_storage_mutate_tasks_nanos_per_event 0.0
> mem_storage_mutate_tasks_nanos_total 149943165
> mem_storage_mutate_tasks_nanos_total_per_sec 0.0
> mem_storage_save_tasks_events 306
> mem_storage_save_tasks_events_per_sec 0.0
> mem_storage_save_tasks_nanos_per_event 0.0
> mem_storage_save_tasks_nanos_total 165988968
> mem_storage_save_tasks_nanos_total_per_sec 0.0
> mem_storage_weakly_consistent_read_operation_events 1538
> mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
> mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
> mem_storage_weakly_consistent_read_operation_nanos_total 138584844
> mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
> mem_storage_write_operation_events 2932
> mem_storage_write_operation_events_per_sec 0.0
> mem_storage_write_operation_nanos_per_event 0.0
> mem_storage_write_operation_nanos_total 28077154879
> mem_storage_write_operation_nanos_total_per_sec 0.0
> new_instance_id_set 0
> offer_accept_races 0
> old_instance_id_set 0
> outstanding_offers 1
> preemptor_attempts 0
> preemptor_failed_preemptions 0
> preemptor_no_slots_found 0
> preemptor_tasks_preempted 0
> process_cpu_cores_utilized 0.0
> process_cpu_time_nanos 73230000000
> process_max_fd_count 4096
> process_open_fd_count 157
> read_lock_wait_nanos 545041
> resources_allocated_quota_cpu 0
> resources_allocated_quota_disk_gb 0
> resources_allocated_quota_ram_gb 0
> resources_dedicated_consumed_cpu 0
> resources_dedicated_consumed_disk_gb 0
> resources_dedicated_consumed_ram_gb 0
> resources_free_pool_consumed_cpu 1
> resources_free_pool_consumed_disk_gb 0
> resources_free_pool_consumed_ram_gb 0
> resources_quota_consumed_cpu 0
> resources_quota_consumed_disk_gb 0
> resources_quota_consumed_ram_gb 0
> resources_total_consumed_cpu 1
> resources_total_consumed_disk_gb 0
> resources_total_consumed_ram_gb 0
> schedule_attempts_failed 0
> schedule_attempts_fired 135
> schedule_queue_size 1
> scheduler_backup_failed 0
> scheduler_backup_success 10
> scheduler_driver_kill_failures 0
> scheduler_failed_offers 0
> scheduler_framework_disconnects 0
> scheduler_framework_reregisters 0
> scheduler_illegal_task_state_transitions 200
> scheduler_log_bad_frames_read 0
> scheduler_log_bytes_read 292813
> scheduler_log_bytes_written 1080061
> scheduler_log_deflated_entries_read 1
> scheduler_log_entries_read 175
> scheduler_log_entries_written 555
> scheduler_log_native_append_events 556
> scheduler_log_native_append_events_per_sec 0.0
> scheduler_log_native_append_failures 0
> scheduler_log_native_append_nanos_per_event 0.0
> scheduler_log_native_append_nanos_total 25082628348
> scheduler_log_native_append_nanos_total_per_sec 0.0
> scheduler_log_native_append_timeouts 0
> scheduler_log_native_native_entries_skipped 1
> scheduler_log_native_read_events 176
> scheduler_log_native_read_events_per_sec 0.0
> scheduler_log_native_read_failures 0
> scheduler_log_native_read_nanos_per_event 0.0
> scheduler_log_native_read_nanos_total 73255842
> scheduler_log_native_read_nanos_total_per_sec 0.0
> scheduler_log_native_read_timeouts 0
> scheduler_log_native_truncate_events 10
> scheduler_log_native_truncate_events_per_sec 0.0
> scheduler_log_native_truncate_failures 0
> scheduler_log_native_truncate_nanos_per_event 0.0
> scheduler_log_native_truncate_nanos_total 276600750
> scheduler_log_native_truncate_nanos_total_per_sec 0.0
> scheduler_log_native_truncate_timeouts 0
> scheduler_log_recover_events 1
> scheduler_log_recover_events_per_sec 0.0
> scheduler_log_recover_nanos_per_event 0.0
> scheduler_log_recover_nanos_total 565333612
> scheduler_log_recover_nanos_total_per_sec 0.0
> scheduler_log_save_framework_id_events 2
> scheduler_log_save_framework_id_events_per_sec 0.0
> scheduler_log_save_framework_id_nanos_per_event 0.0
> scheduler_log_save_framework_id_nanos_total 437292
> scheduler_log_save_framework_id_nanos_total_per_sec 0.0
> scheduler_log_snapshot_events 10
> scheduler_log_snapshot_events_per_sec 0.0
> scheduler_log_snapshot_nanos_per_event 0.0
> scheduler_log_snapshot_nanos_total 961465418
> scheduler_log_snapshot_nanos_total_per_sec 0.0
> scheduler_log_snapshot_persist_events 10
> scheduler_log_snapshot_persist_events_per_sec 0.0
> scheduler_log_snapshot_persist_nanos_per_event 0.0
> scheduler_log_snapshot_persist_nanos_total 840444765
> scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
> scheduler_log_snapshots 10
> scheduler_log_tasks_mutate_events 677
> scheduler_log_tasks_mutate_events_per_sec 0.0
> scheduler_log_tasks_mutate_nanos_per_event 0.0
> scheduler_log_tasks_mutate_nanos_total 182346727
> scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
> scheduler_log_tasks_remove_events 179
> scheduler_log_tasks_remove_events_per_sec 0.0
> scheduler_log_tasks_remove_nanos_per_event 0.0
> scheduler_log_tasks_remove_nanos_total 13588163
> scheduler_log_tasks_remove_nanos_total_per_sec 0.0
> scheduler_log_tasks_save_events 305
> scheduler_log_tasks_save_events_per_sec 0.0
> scheduler_log_tasks_save_nanos_per_event 0.0
> scheduler_log_tasks_save_nanos_total 114479852
> scheduler_log_tasks_save_nanos_total_per_sec 0.0
> scheduler_log_un_snapshotted_transactions 36
> scheduler_lost_executors 0
> scheduler_max_ASSIGNED_waiting_ms 0
> scheduler_max_KILLING_waiting_ms 0
> scheduler_max_PREEMPTING_waiting_ms 0
> scheduler_max_RESTARTING_waiting_ms 0
> scheduler_resource_offers 397
> scheduler_resource_offers_events 397
> scheduler_resource_offers_events_per_sec 0.0
> scheduler_resource_offers_nanos_per_event 0.0
> scheduler_resource_offers_nanos_total 305282357
> scheduler_resource_offers_nanos_total_per_sec 0.0
> scheduler_save_host_attribute_events 397
> scheduler_save_host_attribute_events_per_sec 0.0
> scheduler_save_host_attribute_nanos_per_event 0.0
> scheduler_save_host_attribute_nanos_total 12466197
> scheduler_save_host_attribute_nanos_total_per_sec 0.0
> scheduler_status_update_events 426
> scheduler_status_update_events_per_sec 0.0
> scheduler_status_update_nanos_per_event 0.0
> scheduler_status_update_nanos_total 19334004116
> scheduler_status_update_nanos_total_per_sec 0.0
> scheduler_status_updates 0
> scheduler_thrift_createJob_events 2
> scheduler_thrift_createJob_events_per_sec 0.0
> scheduler_thrift_createJob_nanos_per_event 0.0
> scheduler_thrift_createJob_nanos_total 54102009
> scheduler_thrift_createJob_nanos_total_per_sec 0.0
> scheduler_thrift_getTasksStatus_events 2
> scheduler_thrift_getTasksStatus_events_per_sec 0.0
> scheduler_thrift_getTasksStatus_nanos_per_event 0.0
> scheduler_thrift_getTasksStatus_nanos_total 5684762
> scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
> scheduler_thrift_getVersion_events 4
> scheduler_thrift_getVersion_events_per_sec 0.0
> scheduler_thrift_getVersion_nanos_per_event 0.0
> scheduler_thrift_getVersion_nanos_total 13854714
> scheduler_thrift_getVersion_nanos_total_per_sec 0.0
> scheduler_thrift_killTasks_events 2
> scheduler_thrift_killTasks_events_per_sec 0.0
> scheduler_thrift_killTasks_nanos_per_event 0.0
> scheduler_thrift_killTasks_nanos_total 186407312
> scheduler_thrift_killTasks_nanos_total_per_sec 0.0
> shard_sanity_check_failures 0
> snapshot_apply_events 1
> snapshot_apply_events_per_sec 0.0
> snapshot_apply_nanos_per_event 0.0
> snapshot_apply_nanos_total 93051831
> snapshot_apply_nanos_total_per_sec 0.0
> snapshot_create_events 10
> snapshot_create_events_per_sec 0.0
> snapshot_create_nanos_per_event 0.0
> snapshot_create_nanos_total 37066472
> snapshot_create_nanos_total_per_sec 0.0
> system_env_GLOG_v 0
> system_env_HOME /home/lychee
> system_env_JAVA_OPTS -server -Xmx1g -Xms1g
> -Djava.library.path=/usr/local/lib
> system_env_LANG en_US.UTF-8
> system_env_LIBPROCESS_IP 192.168.0.18
> system_env_LIBPROCESS_PORT 8083
> system_env_LOGNAME root
> system_env_MAIL /var/mail/root
> system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
> system_env_OLDPWD /usr/local/aurora-scheduler
> system_env_PATH
> /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> system_env_PWD /home/lychee
> system_env_SHELL /bin/bash
> system_env_SHLVL 0
> system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
> system_env_SUDO_GID 1002
> system_env_SUDO_UID 1002
> system_env_SUDO_USER lychee
> system_env_TERM xterm
> system_env_USER root
> system_env_USERNAME root
> system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
> system_free_physical_memory_mb 148
> system_free_swap_mb 8063
> system_load_avg 2.22
> task_queries_all 45
> task_queries_by_id 3668
> task_queries_by_job 183
> task_schedule_attempt_events 135
> task_schedule_attempt_events_per_sec 0.0
> task_schedule_attempt_nanos_per_event 0.0
> task_schedule_attempt_nanos_total 6289005821
> task_schedule_attempt_nanos_total_per_sec 0.0
> task_store_ASSIGNED 0
> task_store_FAILED 0
> task_store_FINISHED 200
> task_store_INIT 0
> task_store_KILLED 0
> task_store_KILLING 0
> task_store_LOST 0
> task_store_PENDING 1
> task_store_PREEMPTING 0
> task_store_RESTARTING 0
> task_store_RUNNING 0
> task_store_STARTING 0
> task_store_THROTTLED 0
> task_store_UNKNOWN 0
> timed_out_tasks 0
> timeout_queue_size 338
> transient_states 0
> uncaught_exceptions 0
> variable_scrape_events 38778
> variable_scrape_events_per_sec 1.0000103021061324
> variable_scrape_micros_per_event 455.0001369564521
> variable_scrape_micros_total 18809481
> variable_scrape_micros_total_per_sec 455.00482441615327
> work_queue_depth 0
> write_lock_wait_nanos 13138744
>
>
>
>
> On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <maximk@twitter.com
> >wrote:
>
> > Hi Yingbo,
> >
> > Do you mind sharing your .aurora file? Perhaps your are requesting
> > Resources or Constraints that are infeasible for your local setup to
> > satisfy?
> >
> > Also, you might want to check that your scheduler is receiving offers
> from
> > mesos master. You could check your scheduler log file to see if any
> offers
> > are coming. Alternatively, you might want to look at
> > "scheduler_resource_offers" stat exposed by the scheduler /vars endpoint
> > (http://<your_local_scheduler_ip:port>/vars).
> >
> > Thanks,
> > Maxim
> >
> >
> >
> > On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> >
> > > hi,
> > > Hope you guys have a fun night at city hall.
> > > While you might be drunk, can anyone help me a little bit on setting
> > up
> > > the local dev environment to debug aurora ?
> > > My current set up.
> > > Machine A: 64bit ubuntu server running mesos-master,
> > > Machine B: 64bit ubuntu server running mesos-slave, thermo
> > > observer server;
> > > Machine C: 32bit ubuntu server running zookeeper.
> > > Machine D: Mac air for developing aurora.
> > >
> > > I tried to run aurora-scheduler on Machine A and also created a
> > hello
> > > job from machine A. the job was submitted successfully. however the job
> > > keeps staying at "PENDING" status. i checked thermo observer from port
> > > 1336, it says "No matching hosts".
> > >
> > > I spent a couple of hours and still can't figure out the reason
> so
> > > I decided to settup the dev environment on my Mac so that I can trace
> the
> > > error in IntelliJ. Can someone give me some hint on that? I am not
> quite
> > > familiar with IntelliJ and java.
> > >
> > >
> > >
> > > best
> > > Yingbo
> >
> >
>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Yingbo Wang <yb...@gmail.com>.
Thanks Maxim.
did you mean hello.aurora?
hello = Process(
name = 'hello',
cmdline = """
echo "ls file " ;
whoami;
hostname;
""")
task = SequentialTask(
processes = [hello],
resources = Resources(cpu = 1.0, ram = 128*MB, disk = 128*MB))
jobs = [Service(
task = task, cluster = 'lychee', role = 'lychee', environment = 'prod',
name = 'hello')]
The slave machine has enough resource to satisfy this request.
I did encounter an issue saying "Constraint Not Satisfied, host". After I
change my hostname of both master and slave machines, this problem seems to
be solved, and now the problem becomes "No matching hosts"
Here's the offers that scheduler received:
[
-
{
- id: "201312200632-302033088-5050-15916-396",
- framework_id: "201312110654-302033088-5050-31688-0000",
- slave_id: "201312170824-302033088-5050-25318-0",
- hostname: "192.168.0.11",
- resources:
[
-
{
- name: "cpus",
- scalar: 7.75
},
-
{
- name: "disk",
- scalar: 917136
},
-
{
- name: "mem",
- scalar: 14814
},
-
{
- name: "ports",
- ranges:
[
- "31000-32000"
]
}
],
- attributes:
[
-
{
- name: "host",
- text: "192.168.0.11"
},
-
{
- name: "rack",
- text: "a"
}
],
- executor_ids:
[
-
"thermos-1387559509785-lychee-prod-hello-0-ea46f1c4-4619-49c1-94b5-7d09e1238b7d"
]
}
]
and the /vars are:
async_tasks_completed 393
both_instance_ids_set 0
build_date null
build_git_branchname null
build_git_revision null
build_git_revision_number null
build_git_tag null
build_machine null
build_path null
build_time null
build_timestamp null
build_user_name null
cron_job_launch_failures 0
cron_jobs_triggered 0
cron_num_pending_runs 0
empty_slots_large 0
empty_slots_medium 1
empty_slots_small 7
empty_slots_xlarge 0
framework_registered 1
http_200_responses_events 22
http_200_responses_events_per_sec 0.0
http_200_responses_nanos_per_event 0.0
http_200_responses_nanos_total 981542679
http_200_responses_nanos_total_per_sec 0.0
instance_ids_inconsistent 0
jvm_available_processors 4
jvm_class_loaded_count 6658
jvm_class_total_loaded_count 6658
jvm_class_unloaded_count 0
jvm_gc_PS_MarkSweep_collection_count 0
jvm_gc_PS_MarkSweep_collection_time_ms 0
jvm_gc_PS_Scavenge_collection_count 7
jvm_gc_PS_Scavenge_collection_time_ms 534
jvm_gc_collection_count 7
jvm_gc_collection_time_ms 534
jvm_input_arguments [-Xmx1g, -Xms1g, -Djava.library.path=/usr/local/lib]
jvm_memory_free_mb 688
jvm_memory_heap_mb_committed 968
jvm_memory_heap_mb_max 968
jvm_memory_heap_mb_used 279
jvm_memory_max_mb 968
jvm_memory_mb_total 968
jvm_memory_non_heap_mb_committed 39
jvm_memory_non_heap_mb_max 130
jvm_memory_non_heap_mb_used 38
jvm_prop_awt_toolkit sun.awt.X11.XToolkit
jvm_prop_file_encoding UTF-8
jvm_prop_file_encoding_pkg sun.io
jvm_prop_file_separator /
jvm_prop_java_awt_graphicsenv sun.awt.X11GraphicsEnvironment
jvm_prop_java_awt_printerjob sun.print.PSPrinterJob
jvm_prop_java_class_path
/usr/local/aurora-scheduler/lib/incubator-aurora.jar:/usr/local/aurora-scheduler/lib/aopalliance-1.0.jar:/usr/local/aurora-scheduler/lib/jsr305-1.3.9.jar:/usr/local/aurora-scheduler/lib/guava-14.0.1.jar:/usr/local/aurora-scheduler/lib/guice-3.0.jar:/usr/local/aurora-scheduler/lib/protobuf-java-2.4.1.jar:/usr/local/aurora-scheduler/lib/jersey-core-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-json-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-server-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-servlet-1.12.jar:/usr/local/aurora-scheduler/lib/jersey-guice-1.12.jar:/usr/local/aurora-scheduler/lib/commons-lang-2.5.jar:/usr/local/aurora-scheduler/lib/javax.inject-1.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5.jar:/usr/local/aurora-scheduler/lib/log4j-1.2.17.jar:/usr/local/aurora-scheduler/lib/stringtemplate-3.2.1.jar:/usr/local/aurora-scheduler/lib/mesos-0.15.0-rc3.jar:/usr/local/aurora-scheduler/lib/libthrift-0.9.1.jar:/usr/local/aurora-scheduler/lib/zookeeper-3.3.4.jar:/usr/local/aurora-scheduler/lib/slf4j-api-1.6.1.jar:/usr/local/aurora-scheduler/lib/slf4j-jdk14-1.6.1.jar:/usr/local/aurora-scheduler/lib/log4j-0.0.3.jar:/usr/local/aurora-scheduler/lib/bootstrap-0.0.1.jar:/usr/local/aurora-scheduler/lib/jquery-0.0.1.jar:/usr/local/aurora-scheduler/lib/client-flagged-0.0.2.jar:/usr/local/aurora-scheduler/lib/client-0.0.2.jar:/usr/local/aurora-scheduler/lib/candidate-0.0.48.jar:/usr/local/aurora-scheduler/lib/client-0.0.40.jar:/usr/local/aurora-scheduler/lib/group-0.0.51.jar:/usr/local/aurora-scheduler/lib/server-set-1.0.52.jar:/usr/local/aurora-scheduler/lib/singleton-service-0.0.65.jar:/usr/local/aurora-scheduler/lib/application-http-0.0.49.jar:/usr/local/aurora-scheduler/lib/application-module-http-0.0.46.jar:/usr/local/aurora-scheduler/lib/application-module-lifecycle-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-module-log-0.0.47.jar:/usr/local/aurora-scheduler/lib/application-module-stats-0.0.39.jar:/usr/local/aurora-scheduler/lib/application-0.0.65.jar:/usr/local/aurora-scheduler/lib/args-0.1.71.jar:/usr/local/aurora-scheduler/lib/base-0.0.72.jar:/usr/local/aurora-scheduler/lib/collections-0.0.59.jar:/usr/local/aurora-scheduler/lib/dynamic-host-set-0.0.35.jar:/usr/local/aurora-scheduler/lib/inject-timed-0.0.8.jar:/usr/local/aurora-scheduler/lib/inject-0.0.29.jar:/usr/local/aurora-scheduler/lib/logging-0.0.49.jar:/usr/local/aurora-scheduler/lib/net-http-filters-0.0.3.jar:/usr/local/aurora-scheduler/lib/net-util-0.0.65.jar:/usr/local/aurora-scheduler/lib/quantity-0.0.56.jar:/usr/local/aurora-scheduler/lib/stats-0.0.81.jar:/usr/local/aurora-scheduler/lib/testing-easymock-0.0.1.jar:/usr/local/aurora-scheduler/lib/thrift-0.0.61.jar:/usr/local/aurora-scheduler/lib/util-system-mocks-0.0.55.jar:/usr/local/aurora-scheduler/lib/util-templating-0.0.17.jar:/usr/local/aurora-scheduler/lib/util-testing-0.0.6.jar:/usr/local/aurora-scheduler/lib/util-0.0.83.jar:/usr/local/aurora-scheduler/lib/zookeeper-testing-0.0.37.jar:/usr/local/aurora-scheduler/lib/cglib-2.2.1-v20090111.jar:/usr/local/aurora-scheduler/lib/jettison-1.1.jar:/usr/local/aurora-scheduler/lib/jaxb-impl-2.2.3-1.jar:/usr/local/aurora-scheduler/lib/jackson-core-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-mapper-asl-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-jaxrs-1.9.2.jar:/usr/local/aurora-scheduler/lib/jackson-xc-1.9.2.jar:/usr/local/aurora-scheduler/lib/asm-3.1.jar:/usr/local/aurora-scheduler/lib/guice-servlet-3.0.jar:/usr/local/aurora-scheduler/lib/antlr-2.7.7.jar:/usr/local/aurora-scheduler/lib/commons-lang3-3.1.jar:/usr/local/aurora-scheduler/lib/httpclient-4.2.5.jar:/usr/local/aurora-scheduler/lib/httpcore-4.2.4.jar:/usr/local/aurora-scheduler/lib/jline-0.9.94.jar:/usr/local/aurora-scheduler/lib/lock-0.0.25.jar:/usr/local/aurora-scheduler/lib/map-0.0.34.jar:/usr/local/aurora-scheduler/lib/node-0.0.34.jar:/usr/local/aurora-scheduler/lib/partitioner-0.0.34.jar:/usr/local/aurora-scheduler/lib/application-action-0.0.60.jar:/usr/local/aurora-scheduler/lib/net-http-0.0.37.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-0.0.61.jar:/usr/local/aurora-scheduler/lib/guice-multibindings-3.0.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-time-series-0.0.42.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-pprof-0.0.18.jar:/usr/local/aurora-scheduler/lib/stats-time-series-0.0.47.jar:/usr/local/aurora-scheduler/lib/stats-jvm-0.0.41.jar:/usr/local/aurora-scheduler/lib/commons-io-2.1.jar:/usr/local/aurora-scheduler/lib/commons-codec-1.6.jar:/usr/local/aurora-scheduler/lib/easymock-3.1.jar:/usr/local/aurora-scheduler/lib/junit-dep-4.10.jar:/usr/local/aurora-scheduler/lib/hamcrest-core-1.2.jar:/usr/local/aurora-scheduler/lib/test-libraries-for-java-1.1.1.jar:/usr/local/aurora-scheduler/lib/net-pool-0.0.53.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0.1.jar:/usr/local/aurora-scheduler/lib/jaxb-api-2.2.2.jar:/usr/local/aurora-scheduler/lib/junit-3.8.1.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-text-0.0.27.jar:/usr/local/aurora-scheduler/lib/jetty-6.1.25.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-params-0.0.10.jar:/usr/local/aurora-scheduler/lib/net-http-handlers-string-template-0.0.42.jar:/usr/local/aurora-scheduler/lib/gson-2.2.2.jar:/usr/local/aurora-scheduler/lib/util-core-6.3.8.jar:/usr/local/aurora-scheduler/lib/util-jvm-6.3.8.jar:/usr/local/aurora-scheduler/lib/cglib-nodep-2.2.2.jar:/usr/local/aurora-scheduler/lib/objenesis-1.2.jar:/usr/local/aurora-scheduler/lib/stax-api-1.0-2.jar:/usr/local/aurora-scheduler/lib/activation-1.1.jar:/usr/local/aurora-scheduler/lib/jetty-util-6.1.25.jar:/usr/local/aurora-scheduler/lib/servlet-api-2.5-20081211.jar:/usr/local/aurora-scheduler/lib/scala-library-2.9.2.jar:/usr/local/aurora-scheduler/lib/joda-time-2.1.jar:/usr/local/aurora-scheduler/lib/joda-convert-1.2.jar:/usr/local/aurora-scheduler/lib/args-apt-0.0.45.jar:/usr/local/aurora-scheduler/lib/args-core-0.0.31.jar:/usr/local/aurora-scheduler/lib/io-json-0.0.34.jar:/usr/local/aurora-scheduler/lib/io-thrift-0.0.42.jar:/usr/local/aurora-scheduler/lib/stat-registry-0.0.21.jar:/usr/local/aurora-scheduler/lib/stat-0.0.23.jar:/usr/local/aurora-scheduler/lib/stats-provider-0.0.48.jar:/usr/local/aurora-scheduler/lib/util-sampler-0.0.45.jar:/usr/local/aurora-scheduler/lib/commons-logging-1.1.1.jar:/usr/local/aurora-scheduler/lib/io-0.0.44.jar:/usr/local/aurora-scheduler/lib/service-thrift-1.0.25.jar:/usr/local/aurora-scheduler/lib/application-module-applauncher-0.0.44.jar:/usr/local/aurora-scheduler/lib/util-executor-service-shutdown-0.0.41.jar:/usr/local/aurora-scheduler/lib/jdk-logging-0.0.36.jar
jvm_prop_java_class_version 51.0
jvm_prop_java_endorsed_dirs /usr/lib/jvm/java-7-oracle/jre/lib/endorsed
jvm_prop_java_ext_dirs
/usr/lib/jvm/java-7-oracle/jre/lib/ext:/usr/java/packages/lib/ext
jvm_prop_java_home /usr/lib/jvm/java-7-oracle/jre
jvm_prop_java_io_tmpdir /tmp
jvm_prop_java_library_path /usr/local/lib
jvm_prop_java_runtime_name Java(TM) SE Runtime Environment
jvm_prop_java_runtime_version 1.7.0_45-b18
jvm_prop_java_specification_name Java Platform API Specification
jvm_prop_java_specification_vendor Oracle Corporation
jvm_prop_java_specification_version 1.7
jvm_prop_java_vendor Oracle Corporation
jvm_prop_java_vendor_url http://java.oracle.com/
jvm_prop_java_vendor_url_bug http://bugreport.sun.com/bugreport/
jvm_prop_java_version 1.7.0_45
jvm_prop_java_vm_info mixed mode
jvm_prop_java_vm_name Java HotSpot(TM) 64-Bit Server VM
jvm_prop_java_vm_specification_name Java Virtual Machine Specification
jvm_prop_java_vm_specification_vendor Oracle Corporation
jvm_prop_java_vm_specification_version 1.7
jvm_prop_java_vm_vendor Oracle Corporation
jvm_prop_java_vm_version 24.45-b08
jvm_prop_line_separator
jvm_prop_os_arch amd64
jvm_prop_os_name Linux
jvm_prop_os_version 3.5.0-44-generic
jvm_prop_path_separator :
jvm_prop_sun_arch_data_model 64
jvm_prop_sun_boot_class_path
/usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes
jvm_prop_sun_boot_library_path /usr/lib/jvm/java-7-oracle/jre/lib/amd64
jvm_prop_sun_cpu_endian little
jvm_prop_sun_cpu_isalist
jvm_prop_sun_io_unicode_encoding UnicodeLittle
jvm_prop_sun_java_command
com.twitter.aurora.scheduler.app.SchedulerMain -cluster_name=lychee
-http_port=8081 -thrift_port=8082 -native_log_quorum_size=1
-zk_endpoints=192.168.0.13:2181
-mesos_master_address=zk://192.168.0.13:2181/mesos/master
-serverset_path=/aurora/scheduler
-native_log_zk_group_path=/aurora/replicated-log
-native_log_file_path=/usr/local/aurora-scheduler/db
-backup_dir=/usr/local/aurora-scheduler/backups
-thermos_executor_path=/usr/local/bin/thermos_executor
-gc_executor_path=/usr/local/bin/gc_executor -vlog=INFO -logtostderr
jvm_prop_sun_java_launcher SUN_STANDARD
jvm_prop_sun_jnu_encoding UTF-8
jvm_prop_sun_management_compiler HotSpot 64-Bit Tiered Compilers
jvm_prop_sun_os_patch_level unknown
jvm_prop_user_country US
jvm_prop_user_dir /home/lychee
jvm_prop_user_home /root
jvm_prop_user_language en
jvm_prop_user_name root
jvm_prop_user_timezone Etc/UTC
jvm_threads_active 21
jvm_threads_daemon 12
jvm_threads_peak 25
jvm_threads_started 867
jvm_time_ms 1387559947214
jvm_uptime_secs 38781
log_compressed_entry_bytes_saved 2537635
logging_dir /root
mem_storage_consistent_read_operation_events 619
mem_storage_consistent_read_operation_events_per_sec 0.0
mem_storage_consistent_read_operation_nanos_per_event 0.0
mem_storage_consistent_read_operation_nanos_total 272565131
mem_storage_consistent_read_operation_nanos_total_per_sec 0.0
mem_storage_delete_all_tasks_events 1
mem_storage_delete_all_tasks_events_per_sec 0.0
mem_storage_delete_all_tasks_nanos_per_event 0.0
mem_storage_delete_all_tasks_nanos_total 2323293
mem_storage_delete_all_tasks_nanos_total_per_sec 0.0
mem_storage_delete_tasks_events 179
mem_storage_delete_tasks_events_per_sec 0.0
mem_storage_delete_tasks_nanos_per_event 0.0
mem_storage_delete_tasks_nanos_total 8638495
mem_storage_delete_tasks_nanos_total_per_sec 0.0
mem_storage_fetch_tasks_events 3219
mem_storage_fetch_tasks_events_per_sec 0.0
mem_storage_fetch_tasks_nanos_per_event 0.0
mem_storage_fetch_tasks_nanos_total 193613049
mem_storage_fetch_tasks_nanos_total_per_sec 0.0
mem_storage_mutate_tasks_events 677
mem_storage_mutate_tasks_events_per_sec 0.0
mem_storage_mutate_tasks_nanos_per_event 0.0
mem_storage_mutate_tasks_nanos_total 149943165
mem_storage_mutate_tasks_nanos_total_per_sec 0.0
mem_storage_save_tasks_events 306
mem_storage_save_tasks_events_per_sec 0.0
mem_storage_save_tasks_nanos_per_event 0.0
mem_storage_save_tasks_nanos_total 165988968
mem_storage_save_tasks_nanos_total_per_sec 0.0
mem_storage_weakly_consistent_read_operation_events 1538
mem_storage_weakly_consistent_read_operation_events_per_sec 0.0
mem_storage_weakly_consistent_read_operation_nanos_per_event 0.0
mem_storage_weakly_consistent_read_operation_nanos_total 138584844
mem_storage_weakly_consistent_read_operation_nanos_total_per_sec 0.0
mem_storage_write_operation_events 2932
mem_storage_write_operation_events_per_sec 0.0
mem_storage_write_operation_nanos_per_event 0.0
mem_storage_write_operation_nanos_total 28077154879
mem_storage_write_operation_nanos_total_per_sec 0.0
new_instance_id_set 0
offer_accept_races 0
old_instance_id_set 0
outstanding_offers 1
preemptor_attempts 0
preemptor_failed_preemptions 0
preemptor_no_slots_found 0
preemptor_tasks_preempted 0
process_cpu_cores_utilized 0.0
process_cpu_time_nanos 73230000000
process_max_fd_count 4096
process_open_fd_count 157
read_lock_wait_nanos 545041
resources_allocated_quota_cpu 0
resources_allocated_quota_disk_gb 0
resources_allocated_quota_ram_gb 0
resources_dedicated_consumed_cpu 0
resources_dedicated_consumed_disk_gb 0
resources_dedicated_consumed_ram_gb 0
resources_free_pool_consumed_cpu 1
resources_free_pool_consumed_disk_gb 0
resources_free_pool_consumed_ram_gb 0
resources_quota_consumed_cpu 0
resources_quota_consumed_disk_gb 0
resources_quota_consumed_ram_gb 0
resources_total_consumed_cpu 1
resources_total_consumed_disk_gb 0
resources_total_consumed_ram_gb 0
schedule_attempts_failed 0
schedule_attempts_fired 135
schedule_queue_size 1
scheduler_backup_failed 0
scheduler_backup_success 10
scheduler_driver_kill_failures 0
scheduler_failed_offers 0
scheduler_framework_disconnects 0
scheduler_framework_reregisters 0
scheduler_illegal_task_state_transitions 200
scheduler_log_bad_frames_read 0
scheduler_log_bytes_read 292813
scheduler_log_bytes_written 1080061
scheduler_log_deflated_entries_read 1
scheduler_log_entries_read 175
scheduler_log_entries_written 555
scheduler_log_native_append_events 556
scheduler_log_native_append_events_per_sec 0.0
scheduler_log_native_append_failures 0
scheduler_log_native_append_nanos_per_event 0.0
scheduler_log_native_append_nanos_total 25082628348
scheduler_log_native_append_nanos_total_per_sec 0.0
scheduler_log_native_append_timeouts 0
scheduler_log_native_native_entries_skipped 1
scheduler_log_native_read_events 176
scheduler_log_native_read_events_per_sec 0.0
scheduler_log_native_read_failures 0
scheduler_log_native_read_nanos_per_event 0.0
scheduler_log_native_read_nanos_total 73255842
scheduler_log_native_read_nanos_total_per_sec 0.0
scheduler_log_native_read_timeouts 0
scheduler_log_native_truncate_events 10
scheduler_log_native_truncate_events_per_sec 0.0
scheduler_log_native_truncate_failures 0
scheduler_log_native_truncate_nanos_per_event 0.0
scheduler_log_native_truncate_nanos_total 276600750
scheduler_log_native_truncate_nanos_total_per_sec 0.0
scheduler_log_native_truncate_timeouts 0
scheduler_log_recover_events 1
scheduler_log_recover_events_per_sec 0.0
scheduler_log_recover_nanos_per_event 0.0
scheduler_log_recover_nanos_total 565333612
scheduler_log_recover_nanos_total_per_sec 0.0
scheduler_log_save_framework_id_events 2
scheduler_log_save_framework_id_events_per_sec 0.0
scheduler_log_save_framework_id_nanos_per_event 0.0
scheduler_log_save_framework_id_nanos_total 437292
scheduler_log_save_framework_id_nanos_total_per_sec 0.0
scheduler_log_snapshot_events 10
scheduler_log_snapshot_events_per_sec 0.0
scheduler_log_snapshot_nanos_per_event 0.0
scheduler_log_snapshot_nanos_total 961465418
scheduler_log_snapshot_nanos_total_per_sec 0.0
scheduler_log_snapshot_persist_events 10
scheduler_log_snapshot_persist_events_per_sec 0.0
scheduler_log_snapshot_persist_nanos_per_event 0.0
scheduler_log_snapshot_persist_nanos_total 840444765
scheduler_log_snapshot_persist_nanos_total_per_sec 0.0
scheduler_log_snapshots 10
scheduler_log_tasks_mutate_events 677
scheduler_log_tasks_mutate_events_per_sec 0.0
scheduler_log_tasks_mutate_nanos_per_event 0.0
scheduler_log_tasks_mutate_nanos_total 182346727
scheduler_log_tasks_mutate_nanos_total_per_sec 0.0
scheduler_log_tasks_remove_events 179
scheduler_log_tasks_remove_events_per_sec 0.0
scheduler_log_tasks_remove_nanos_per_event 0.0
scheduler_log_tasks_remove_nanos_total 13588163
scheduler_log_tasks_remove_nanos_total_per_sec 0.0
scheduler_log_tasks_save_events 305
scheduler_log_tasks_save_events_per_sec 0.0
scheduler_log_tasks_save_nanos_per_event 0.0
scheduler_log_tasks_save_nanos_total 114479852
scheduler_log_tasks_save_nanos_total_per_sec 0.0
scheduler_log_un_snapshotted_transactions 36
scheduler_lost_executors 0
scheduler_max_ASSIGNED_waiting_ms 0
scheduler_max_KILLING_waiting_ms 0
scheduler_max_PREEMPTING_waiting_ms 0
scheduler_max_RESTARTING_waiting_ms 0
scheduler_resource_offers 397
scheduler_resource_offers_events 397
scheduler_resource_offers_events_per_sec 0.0
scheduler_resource_offers_nanos_per_event 0.0
scheduler_resource_offers_nanos_total 305282357
scheduler_resource_offers_nanos_total_per_sec 0.0
scheduler_save_host_attribute_events 397
scheduler_save_host_attribute_events_per_sec 0.0
scheduler_save_host_attribute_nanos_per_event 0.0
scheduler_save_host_attribute_nanos_total 12466197
scheduler_save_host_attribute_nanos_total_per_sec 0.0
scheduler_status_update_events 426
scheduler_status_update_events_per_sec 0.0
scheduler_status_update_nanos_per_event 0.0
scheduler_status_update_nanos_total 19334004116
scheduler_status_update_nanos_total_per_sec 0.0
scheduler_status_updates 0
scheduler_thrift_createJob_events 2
scheduler_thrift_createJob_events_per_sec 0.0
scheduler_thrift_createJob_nanos_per_event 0.0
scheduler_thrift_createJob_nanos_total 54102009
scheduler_thrift_createJob_nanos_total_per_sec 0.0
scheduler_thrift_getTasksStatus_events 2
scheduler_thrift_getTasksStatus_events_per_sec 0.0
scheduler_thrift_getTasksStatus_nanos_per_event 0.0
scheduler_thrift_getTasksStatus_nanos_total 5684762
scheduler_thrift_getTasksStatus_nanos_total_per_sec 0.0
scheduler_thrift_getVersion_events 4
scheduler_thrift_getVersion_events_per_sec 0.0
scheduler_thrift_getVersion_nanos_per_event 0.0
scheduler_thrift_getVersion_nanos_total 13854714
scheduler_thrift_getVersion_nanos_total_per_sec 0.0
scheduler_thrift_killTasks_events 2
scheduler_thrift_killTasks_events_per_sec 0.0
scheduler_thrift_killTasks_nanos_per_event 0.0
scheduler_thrift_killTasks_nanos_total 186407312
scheduler_thrift_killTasks_nanos_total_per_sec 0.0
shard_sanity_check_failures 0
snapshot_apply_events 1
snapshot_apply_events_per_sec 0.0
snapshot_apply_nanos_per_event 0.0
snapshot_apply_nanos_total 93051831
snapshot_apply_nanos_total_per_sec 0.0
snapshot_create_events 10
snapshot_create_events_per_sec 0.0
snapshot_create_nanos_per_event 0.0
snapshot_create_nanos_total 37066472
snapshot_create_nanos_total_per_sec 0.0
system_env_GLOG_v 0
system_env_HOME /home/lychee
system_env_JAVA_OPTS -server -Xmx1g -Xms1g -Djava.library.path=/usr/local/lib
system_env_LANG en_US.UTF-8
system_env_LIBPROCESS_IP 192.168.0.18
system_env_LIBPROCESS_PORT 8083
system_env_LOGNAME root
system_env_MAIL /var/mail/root
system_env_NLSPATH /usr/dt/lib/nls/msg/%L/%N.cat
system_env_OLDPWD /usr/local/aurora-scheduler
system_env_PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
system_env_PWD /home/lychee
system_env_SHELL /bin/bash
system_env_SHLVL 0
system_env_SUDO_COMMAND /etc/rc.aurora_scheduler.local
system_env_SUDO_GID 1002
system_env_SUDO_UID 1002
system_env_SUDO_USER lychee
system_env_TERM xterm
system_env_USER root
system_env_USERNAME root
system_env_XFILESEARCHPATH /usr/dt/app-defaults/%L/Dt
system_free_physical_memory_mb 148
system_free_swap_mb 8063
system_load_avg 2.22
task_queries_all 45
task_queries_by_id 3668
task_queries_by_job 183
task_schedule_attempt_events 135
task_schedule_attempt_events_per_sec 0.0
task_schedule_attempt_nanos_per_event 0.0
task_schedule_attempt_nanos_total 6289005821
task_schedule_attempt_nanos_total_per_sec 0.0
task_store_ASSIGNED 0
task_store_FAILED 0
task_store_FINISHED 200
task_store_INIT 0
task_store_KILLED 0
task_store_KILLING 0
task_store_LOST 0
task_store_PENDING 1
task_store_PREEMPTING 0
task_store_RESTARTING 0
task_store_RUNNING 0
task_store_STARTING 0
task_store_THROTTLED 0
task_store_UNKNOWN 0
timed_out_tasks 0
timeout_queue_size 338
transient_states 0
uncaught_exceptions 0
variable_scrape_events 38778
variable_scrape_events_per_sec 1.0000103021061324
variable_scrape_micros_per_event 455.0001369564521
variable_scrape_micros_total 18809481
variable_scrape_micros_total_per_sec 455.00482441615327
work_queue_depth 0
write_lock_wait_nanos 13138744
On Fri, Dec 20, 2013 at 9:07 AM, Maxim Khutornenko <ma...@twitter.com>wrote:
> Hi Yingbo,
>
> Do you mind sharing your .aurora file? Perhaps your are requesting
> Resources or Constraints that are infeasible for your local setup to
> satisfy?
>
> Also, you might want to check that your scheduler is receiving offers from
> mesos master. You could check your scheduler log file to see if any offers
> are coming. Alternatively, you might want to look at
> "scheduler_resource_offers" stat exposed by the scheduler /vars endpoint
> (http://<your_local_scheduler_ip:port>/vars).
>
> Thanks,
> Maxim
>
>
>
> On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
>
> > hi,
> > Hope you guys have a fun night at city hall.
> > While you might be drunk, can anyone help me a little bit on setting
> up
> > the local dev environment to debug aurora ?
> > My current set up.
> > Machine A: 64bit ubuntu server running mesos-master,
> > Machine B: 64bit ubuntu server running mesos-slave, thermo
> > observer server;
> > Machine C: 32bit ubuntu server running zookeeper.
> > Machine D: Mac air for developing aurora.
> >
> > I tried to run aurora-scheduler on Machine A and also created a
> hello
> > job from machine A. the job was submitted successfully. however the job
> > keeps staying at "PENDING" status. i checked thermo observer from port
> > 1336, it says "No matching hosts".
> >
> > I spent a couple of hours and still can't figure out the reason so
> > I decided to settup the dev environment on my Mac so that I can trace the
> > error in IntelliJ. Can someone give me some hint on that? I am not quite
> > familiar with IntelliJ and java.
> >
> >
> >
> > best
> > Yingbo
>
>
Re: Instruction on setting up dev environment on my Mac machine
Posted by Maxim Khutornenko <ma...@twitter.com>.
Hi Yingbo,
Do you mind sharing your .aurora file? Perhaps your are requesting Resources or Constraints that are infeasible for your local setup to satisfy?
Also, you might want to check that your scheduler is receiving offers from mesos master. You could check your scheduler log file to see if any offers are coming. Alternatively, you might want to look at "scheduler_resource_offers" stat exposed by the scheduler /vars endpoint (http://<your_local_scheduler_ip:port>/vars).
Thanks,
Maxim
On Dec 19, 2013, at 11:20 PM, Yingbo Wang <yb...@gmail.com> wrote:
> hi,
> Hope you guys have a fun night at city hall.
> While you might be drunk, can anyone help me a little bit on setting up
> the local dev environment to debug aurora ?
> My current set up.
> Machine A: 64bit ubuntu server running mesos-master,
> Machine B: 64bit ubuntu server running mesos-slave, thermo
> observer server;
> Machine C: 32bit ubuntu server running zookeeper.
> Machine D: Mac air for developing aurora.
>
> I tried to run aurora-scheduler on Machine A and also created a hello
> job from machine A. the job was submitted successfully. however the job
> keeps staying at "PENDING" status. i checked thermo observer from port
> 1336, it says "No matching hosts".
>
> I spent a couple of hours and still can't figure out the reason so
> I decided to settup the dev environment on my Mac so that I can trace the
> error in IntelliJ. Can someone give me some hint on that? I am not quite
> familiar with IntelliJ and java.
>
>
>
> best
> Yingbo