You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Ben Mahler <be...@gmail.com> on 2013/03/02 00:58:55 UTC
Review Request: Quieter allocator logging.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/
-----------------------------------------------------------
Review request for mesos, Benjamin Hindman and Vinod Kone.
Description
-------
This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
Before:
31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
After:
31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
Diffs
-----
src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
Diff: https://reviews.apache.org/r/9711/diff/
Testing
-------
Thanks,
Ben Mahler
Re: Review Request: Quieter allocator logging.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/#review17288
-----------------------------------------------------------
Ship it!
Ship It!
- Vinod Kone
On March 1, 2013, 11:58 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9711/
> -----------------------------------------------------------
>
> (Updated March 1, 2013, 11:58 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Description
> -------
>
> This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
>
> I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
>
> I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
> I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
> I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
> I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
> I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
>
>
> My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
>
> Before:
> 31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
>
>
> After:
> 31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
>
>
> Diffs
> -----
>
> src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
>
> Diff: https://reviews.apache.org/r/9711/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request: Quieter allocator logging.
Posted by Benjamin Hindman <be...@berkeley.edu>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/#review17358
-----------------------------------------------------------
Ship it!
Ship It!
- Benjamin Hindman
On March 4, 2013, 8:22 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9711/
> -----------------------------------------------------------
>
> (Updated March 4, 2013, 8:22 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Description
> -------
>
> This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
>
> I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
>
> I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
> I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
> I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
> I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
> I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
>
>
> My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
>
> Before:
> 31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
>
>
> After:
> 31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
>
>
> Diffs
> -----
>
> src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
>
> Diff: https://reviews.apache.org/r/9711/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request: Quieter allocator logging.
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/
-----------------------------------------------------------
(Updated March 4, 2013, 8:22 p.m.)
Review request for mesos, Benjamin Hindman and Vinod Kone.
Changes
-------
Updated to trunk.
Description
-------
This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
Before:
31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
After:
31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
Diffs (updated)
-----
src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
Diff: https://reviews.apache.org/r/9711/diff/
Testing
-------
Thanks,
Ben Mahler
Re: Review Request: Quieter allocator logging.
Posted by Ben Mahler <be...@gmail.com>.
> On March 2, 2013, 12:19 a.m., Vinod Kone wrote:
> > src/master/hierarchical_allocator_process.hpp, line 617
> > <https://reviews.apache.org/r/9711/diff/1/?file=264386#file264386line617>
> >
> > I think this could be LOG(INFO) because this is only called when a slave is added?
I'd argue it's redundant (given it gets accompanied by another log line), we can do with the timing log being always here, right?
I0227 23:47:14.574532 1588 hierarchical_allocator_process.hpp:393] Added slave 201302272335-2206671370-5050-1580-6717 (<snip>) with cpus=0.5; mem=1600; disk=6400; ports=[31934-31934] (
and cpus=0.5; mem=1600; disk=6400; ports=[31934-31934] available)
I0227 23:47:14.591006 1588 hierarchical_allocator_process.hpp:617] Performed allocation for slave 201302272335-2206671370-5050-1580-6717 in 16.42ms
- Ben
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/#review17286
-----------------------------------------------------------
On March 1, 2013, 11:58 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9711/
> -----------------------------------------------------------
>
> (Updated March 1, 2013, 11:58 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Description
> -------
>
> This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
>
> I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
>
> I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
> I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
> I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
> I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
> I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
>
>
> My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
>
> Before:
> 31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
>
>
> After:
> 31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
>
>
> Diffs
> -----
>
> src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
>
> Diff: https://reviews.apache.org/r/9711/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ben Mahler
>
>
Re: Review Request: Quieter allocator logging.
Posted by Vinod Kone <vi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/9711/#review17286
-----------------------------------------------------------
src/master/hierarchical_allocator_process.hpp
<https://reviews.apache.org/r/9711/#comment36754>
I think this could be LOG(INFO) because this is only called when a slave is added?
- Vinod Kone
On March 1, 2013, 11:58 p.m., Ben Mahler wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/9711/
> -----------------------------------------------------------
>
> (Updated March 1, 2013, 11:58 p.m.)
>
>
> Review request for mesos, Benjamin Hindman and Vinod Kone.
>
>
> Description
> -------
>
> This only takes care of the noisy allocator timing logs, by moving them to VLOG(1).
>
> I'd also like to try to take care of the log noise from heavily scheduled machines with port fragmentation, check out the output below:
>
> I0301 20:50:20.594537 22294 master.hpp:296] Removing task with resources cpus=0.01; disk=1; mem=1 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:20.595034 22294 hierarchical_allocator_process.hpp:542] Recovered cpus=0.01; disk=1; mem=1 (total allocatable: cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185) on slave 201302272335-1937777162-5050-7468-64 from framework 201103282247-0000000019-0000
> I0301 20:50:21.489853 22289 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.62ms
> I0301 20:50:21.490259 22300 master.hpp:305] Adding offer with resources cpus=12.31; mem=18442; ports=[31001-31004, 31006-31006, 31008-31011, 31013-31013, 31015-31022, 31025-31027, 31030-31037, 31039-31057, 31059-31059, 31061-31061, 31063-31064, 31067-31081, 31083-31106, 31108-31109, 31111-31114, 31116-31126, 31128-31136, 31139-31147, 31149-31150, 31152-31158, 31160-31161, 31163-31166, 31168-31169, 31171-31176, 31179-31184, 31189-31190, 31192-31194, 31196-31196, 31198-31200, 31202-31202, 31204-31208, 31210-31223, 31225-31229, 31231-31232, 31234-31235, 31237-31239, 31241-31244, 31246-31253, 31255-31265, 31267-31268, 31270-31270, 31273-31284, 31286-31290, 31292-31294, 31297-31300, 31302-31304, 31306-31310, 31312-31315, 31317-31324, 31326-31328, 31331-31339, 31343-31343, 31345-31348, 31350-31357, 31359-31359, 31361-31365, 31367-31375, 31377-31381, 31383-31396, 31398-31401, 31403-31403, 31405-31405, 31408-31422, 31424-31434, 31437-31451, 31453-31458, 31460-31464, 31467-31467, 31469-31473, 31476-31486, 31489-31493, 31496-31496, 31498-31498, 31500-31504, 31506-31507, 31509-31510, 31512-31515, 31517-31518, 31520-31528, 31530-31536, 31539-31545, 31548-31550, 31552-31552, 31554-31557, 31559-31560, 31562-31562, 31565-31569, 31571-31577, 31579-31581, 31583-31586, 31590-31590, 31592-31600, 31602-31607, 31609-31612, 31614-31614, 31616-31616, 31618-31621, 31623-31625, 31627-31628, 31630-31630, 31632-31632, 31634-31642, 31644-31666, 31668-31672, 31674-31684, 31686-31686, 31688-31693, 31695-31695, 31697-31702, 31704-31709, 31711-31711, 31713-31717, 31719-31725, 31727-31741, 31743-31743, 31745-31746, 31748-31750, 31752-31759, 31762-31771, 31773-31779, 31781-31781, 31787-31795, 31797-31797, 31800-31806, 31808-31812, 31814-31815, 31817-31823, 31825-31832, 31834-31839, 31841-31842, 31845-31851, 31853-31857, 31861-31864, 31867-31867, 31870-31879, 31881-31881, 31883-31885, 31887-31887, 31889-31889, 31891-31892, 31894-31896, 31898-31898, 31900-31901, 31904-31909, 31911-31911, 31913-31914, 31916-31916, 31918-31927, 31929-31933, 31935-31937, 31939-31943, 31945-31949, 31951-31951, 31953-31957, 31960-31961, 31963-31965, 31967-31969, 31971-31981, 31984-31985, 31988-31993, 31995-32000]; disk=387185 on slave 201302272335-1937777162-5050-7468-64
> I0301 20:50:21.492420 22300 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:21.578040 22294 master.cpp:79] No whitelist given. Advertising offers for all slaves
> I0301 20:50:22.578544 22297 master.hpp:305] Adding offer with resources cpus=11.75; mem=18073; ports=[31001-31001, 31003-31010, 31015-31034, 31036-31040, 31042-31045, 31047-31050, 31054-31058, 31060-31060, 31062-31081, 31083-31086, 31088-31091, 31093-31093, 31095-31096, 31098-31101, 31103-31113, 31115-31115, 31117-31121, 31125-31125, 31127-31131, 31133-31134, 31136-31137, 31139-31154, 31156-31156, 31159-31161, 31164-31167, 31169-31171, 31176-31178, 31180-31180, 31182-31193, 31195-31197, 31200-31202, 31204-31208, 31211-31215, 31218-31221, 31223-31224, 31227-31227, 31229-31229, 31231-31236, 31238-31240, 31243-31257, 31261-31270, 31272-31282, 31284-31289, 31291-31298, 31300-31301, 31303-31306, 31312-31316, 31318-31318, 31320-31322, 31324-31324, 31326-31330, 31332-31335, 31338-31339, 31341-31346, 31348-31349, 31352-31353, 31355-31356, 31358-31362, 31365-31365, 31367-31377, 31379-31387, 31389-31394, 31396-31398, 31400-31409, 31411-31411, 31413-31413, 31415-31416, 31418-31428, 31430-31431, 31434-31446, 31448-31452, 31456-31461, 31463-31464, 31466-31469, 31471-31471, 31474-31474, 31476-31478, 31480-31484, 31486-31486, 31488-31490, 31492-31493, 31495-31498, 31500-31508, 31510-31520, 31522-31523, 31525-31525, 31528-31529, 31531-31539, 31541-31543, 31546-31558, 31560-31560, 31563-31564, 31566-31571, 31573-31575, 31577-31582, 31584-31585, 31588-31592, 31595-31607, 31609-31617, 31619-31628, 31631-31642, 31644-31645, 31648-31649, 31652-31652, 31654-31676, 31678-31708, 31710-31717, 31719-31724, 31727-31727, 31729-31742, 31744-31748, 31750-31752, 31754-31754, 31756-31757, 31759-31765, 31767-31770, 31772-31781, 31783-31789, 31791-31805, 31807-31811, 31813-31814, 31817-31820, 31823-31830, 31832-31833, 31835-31836, 31838-31839, 31841-31846, 31848-31848, 31850-31856, 31858-31861, 31863-31863, 31866-31870, 31872-31873, 31878-31879, 31882-31883, 31886-31891, 31893-31894, 31897-31897, 31899-31911, 31913-31915, 31917-31927, 31929-31936, 31938-31939, 31941-31942, 31944-31947, 31949-31958, 31960-31967, 31969-31978, 31980-31981, 31983-31989, 31991-31995, 31998-32000]; disk=386176 on slave 201302272335-1937777162-5050-7468-61
> I0301 20:50:22.579871 22304 hierarchical_allocator_process.hpp:597] Performed allocation for 9477 slaves in 89.37ms
> I0301 20:50:22.580482 22297 master.cpp:1262] Sending 1 offers to framework 201103282247-0000000019-0000
> W0301 20:50:23.305022 22291 master.cpp:791] Offer 201302280043-2206671370-5050-22288-65505 is no longer valid
>
>
> My first attempt was to examine the difference printing singular port ranges would make, but it's not significant enough:
>
> Before:
> 31000-31000, 31003-31006, 31008-31009, 31015-31016, 31018-31018, 31026-31026, 31035-31036, 31038-31038, 31045-31045, 31048-31048, 31051-31051, 31053-31054, 31058-31060, 31063-31067, 31069-31069, 31071-31071, 31073-31073, 31075-31075, 31077-31078, 31080-31082, 31086-31087, 31091-31091, 31095-31095, 31097-31099, 31102-31102, 31104-31104, 31109-31111, 31113-31115, 31119-31119, 31121-31122, 31125-31125, 31127-31130, 31133-31134, 31136-31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156-31156, 31159-31159, 31164-31166, 31168-31168, 31171-31172, 31177-31180, 31184-31187, 31190-31190, 31192-31193, 31195-31200, 31202-31202, 31207-31207, 31209-31211, 31213-31214, 31216-31216, 31218-31219, 31221-31222, 31224-31225, 31228-31228, 31230-31230, 31234-31234, 31236-31237, 31239-31244, 31249-31250, 31252-31252, 31257-31259, 31262-31263, 31266-31267, 31271-31271, 31276-31276, 31283-31283, 31285-31285, 31287-31287, 31290-31297, 31299-31303, 31307-31307, 31313-31314, 31316-31316, 31318-31321, 31323-31323, 31325-31325, 31328-31328, 31332-31332, 31335-31338, 31340-31342, 31344-31344, 31346-31346, 31348-31349, 31351-31351, 31353-31354, 31356-31356, 31358-31358, 31363-31365, 31371-31371, 31373-31373, 31375-31377, 31380-31381, 31387-31387, 31392-31393, 31398-31399, 31402-31402, 31405-31407, 31409-31409, 31412-31412, 31415-31415, 31418-31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447-31447, 31449-31451, 31454-31454, 31459-31460, 31463-31463, 31471-31473, 31476-31477, 31479-31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504-31504, 31506-31506, 31508-31508, 31510-31511, 31516-31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549-31549, 31553-31553, 31558-31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578-31578, 31580-31580, 31582-31583, 31585-31588, 31590-31590, 31593-31593, 31595-31597, 31599-31600, 31602-31603, 31605-31605, 31608-31610, 31612-31613, 31616-31616, 31623-31624, 31627-31627, 31632-31632, 31634-31634, 31636-31636, 31640-31641, 31643-31643, 31649-31649, 31655-31655, 31657-31660, 31662-31662, 31664-31666, 31670-31673, 31677-31677, 31679-31680, 31684-31686, 31689-31690, 31693-31693, 31695-31697, 31699-31699, 31701-31704, 31707-31707, 31709-31709, 31711-31712, 31714-31714, 31716-31716, 31718-31718, 31723-31724, 31726-31728, 31730-31730, 31734-31734, 31736-31738, 31740-31740, 31742-31743, 31745-31745, 31747-31747, 31749-31750, 31752-31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797-31797, 31799-31801, 31804-31805, 31807-31807, 31809-31809, 31811-31814, 31817-31817, 31820-31820, 31822-31822, 31827-31827, 31829-31831, 31833-31834, 31836-31836, 31838-31838, 31842-31843, 31846-31846, 31848-31849, 31853-31853, 31855-31855, 31859-31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887-31887, 31890-31890, 31894-31895, 31898-31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926-31926, 31928-31929, 31937-31938, 31940-31940, 31942-31943, 31947-31950, 31953-31953, 31955-31956, 31958-31963, 31965-31968, 31973-31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992-31992, 31994-31994, 31999-32000
>
>
> After:
> 31000, 31003-31006, 31008-31009, 31015-31016, 31018, 31026, 31035-31036, 31038, 31045, 31048, 31051, 31053-31054, 31058-31060, 31063-31067, 31069, 31071, 31073, 31075, 31077-31078, 31080-31082, 31086-31087, 31091, 31095, 31097-31099, 31102, 31104, 31109-31111, 31113-31115, 31119, 31121-31122, 31125, 31127-31130, 31133-31134, 31136, 31141-31142, 31144-31145, 31147-31148, 31153-31154, 31156, 31159, 31164-31166, 31168, 31171-31172, 31177-31180, 31184-31187, 31190, 31192-31193, 31195-31200, 31202, 31207, 31209-31211, 31213-31214, 31216, 31218-31219, 31221-31222, 31224-31225, 31228, 31230, 31234, 31236-31237, 31239-31244, 31249-31250, 31252, 31257-31259, 31262-31263, 31266-31267, 31271, 31276, 31283, 31285, 31287, 31290-31297, 31299-31303, 31307, 31313-31314, 31316, 31318-31321, 31323, 31325, 31328, 31332, 31335-31338, 31340-31342, 31344, 31346, 31348-31349, 31351, 31353-31354, 31356, 31358, 31363-31365, 31371, 31373, 31375-31377, 31380-31381, 31387, 31392-31393, 31398-31399, 31402, 31405-31407, 31409, 31412, 31415, 31418, 31420-31425, 31428-31430, 31434-31435, 31439-31441, 31443-31445, 31447, 31449-31451, 31454, 31459-31460, 31463, 31471-31473, 31476-31477, 31479, 31487-31489, 31491-31492, 31495-31497, 31500-31501, 31504, 31506, 31508, 31510-31511, 31516, 31520-31521, 31523-31531, 31533-31534, 31537-31544, 31546-31547, 31549, 31553, 31558, 31560-31561, 31564-31565, 31569-31570, 31572-31576, 31578, 31580, 31582-31583, 31585-31588, 31590, 31593, 31595-31597, 31599-31600, 31602-31603, 31605, 31608-31610, 31612-31613, 31616, 31623-31624, 31627, 31632, 31634, 31636, 31640-31641, 31643, 31649, 31655, 31657-31660, 31662, 31664-31666, 31670-31673, 31677, 31679-31680, 31684-31686, 31689-31690, 31693, 31695-31697, 31699, 31701-31704, 31707, 31709, 31711-31712, 31714, 31716, 31718, 31723-31724, 31726-31728, 31730, 31734, 31736-31738, 31740, 31742-31743, 31745, 31747, 31749-31750, 31752, 31756-31757, 31759-31763, 31766-31771, 31773-31774, 31777-31778, 31786-31787, 31790-31791, 31794-31795, 31797, 31799-31801, 31804-31805, 31807, 31809, 31811-31814, 31817, 31820, 31822, 31827, 31829-31831, 31833-31834, 31836, 31838, 31842-31843, 31846, 31848-31849, 31853, 31855, 31859, 31862-31864, 31866-31867, 31871-31872, 31876-31881, 31887, 31890, 31894-31895, 31898, 31901-31904, 31906-31910, 31912-31913, 31917-31920, 31923-31924, 31926, 31928-31929, 31937-31938, 31940, 31942-31943, 31947-31950, 31953, 31955-31956, 31958-31963, 31965-31968, 31973, 31977-31978, 31980-31981, 31984-31986, 31988-31990, 31992, 31994, 31999-32000,
>
>
> Diffs
> -----
>
> src/master/hierarchical_allocator_process.hpp 33e059c10bb87db88b1fb9a100b7504bc89ac896
>
> Diff: https://reviews.apache.org/r/9711/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Ben Mahler
>
>