You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Alan Conway (JIRA)" <ji...@apache.org> on 2017/10/03 19:33:00 UTC

[jira] [Created] (DISPATCH-846) Memory leaks in ctest with valgrind

Alan Conway created DISPATCH-846:
------------------------------------

             Summary: Memory leaks in ctest with valgrind
                 Key: DISPATCH-846
                 URL: https://issues.apache.org/jira/browse/DISPATCH-846
             Project: Qpid Dispatch
          Issue Type: Bug
          Components: Container
    Affects Versions: 0.8.0
            Reporter: Alan Conway
             Fix For: 0.8.1, 1.0.0


There are several memory leaks reported by ctest with valigrind, example below. The exact set of leaks is not always the same, so there is probably a race condition involved, but I get some leaks on pretty much every run.

Aside: we stared using the gcc/clang sanitizers with proton, they require special build flags but are *way* faster than valgrind's memcheck/helgrind with similar error-detection.

{code}
make && ctest 
[ 70%] Built target qpid-dispatch
[ 78%] Built target unit_tests_size
[ 85%] Built target qdrouterd
[ 93%] Built target unit_tests
[100%] Built target doc_gen
Test project /home/aconway/dispatch/reldbg
      Start  1: unit_tests_size_10000
 1/35 Test  #1: unit_tests_size_10000 .....................   Passed    1.18 sec
      Start  2: unit_tests_size_512
 2/35 Test  #2: unit_tests_size_512 .......................   Passed    1.14 sec
      Start  3: unit_tests_size_10
 3/35 Test  #3: unit_tests_size_10 ........................   Passed    1.18 sec
      Start  4: unit_tests_size_7
 4/35 Test  #4: unit_tests_size_7 .........................   Passed    1.13 sec
      Start  5: unit_tests_size_5
 5/35 Test  #5: unit_tests_size_5 .........................   Passed    1.21 sec
      Start  6: unit_tests_size_3
 6/35 Test  #6: unit_tests_size_3 .........................   Passed    1.20 sec
      Start  7: unit_tests_size_2
 7/35 Test  #7: unit_tests_size_2 .........................   Passed    1.20 sec
      Start  8: unit_tests_size_1
 8/35 Test  #8: unit_tests_size_1 .........................   Passed    1.15 sec
      Start  9: unit_tests
 9/35 Test  #9: unit_tests ................................   Passed    4.47 sec
      Start 10: router_engine_test
10/35 Test #10: router_engine_test ........................   Passed    0.05 sec
      Start 11: management_test
11/35 Test #11: management_test ...........................   Passed    0.04 sec
      Start 12: router_policy_test
12/35 Test #12: router_policy_test ........................   Passed    0.10 sec
      Start 13: system_tests_link_routes
13/35 Test #13: system_tests_link_routes ..................***Failed   24.66 sec
test_aaa_qdmanage_query_link_route (system_tests_link_routes.LinkRouteTest) ... ok
test_bbb_qdstat_link_routes_routerB (system_tests_link_routes.LinkRouteTest) ... FAIL
test_ccc_qdstat_link_routes_routerC (system_tests_link_routes.LinkRouteTest) ... ok
test_close_with_unsettled (system_tests_link_routes.LinkRouteTest) ... ok
test_custom_annotations_match (system_tests_link_routes.LinkRouteTest) ... ok
test_ddd_partial_link_route_match (system_tests_link_routes.LinkRouteTest) ... ok
test_detach_mixed_close (system_tests_link_routes.LinkRouteTest) ... ok
test_detach_without_close (system_tests_link_routes.LinkRouteTest) ... ok
test_dynamic_source (system_tests_link_routes.LinkRouteTest) ... ok
test_dynamic_target (system_tests_link_routes.LinkRouteTest) ... ok
test_full_link_route_match (system_tests_link_routes.LinkRouteTest) ... ok
test_full_link_route_match_1 (system_tests_link_routes.LinkRouteTest) ... ok
test_link_route_pattern_match (system_tests_link_routes.LinkRouteTest) ... ok
test_link_route_terminus_address (system_tests_link_routes.LinkRouteTest) ... ok
test_partial_link_route_match_1 (system_tests_link_routes.LinkRouteTest) ... ok
test_www_drain_support_all_messages (system_tests_link_routes.LinkRouteTest) ... ok
test_www_drain_support_no_messages (system_tests_link_routes.LinkRouteTest) ... ok
test_www_drain_support_no_more_messages (system_tests_link_routes.LinkRouteTest) ... ok
test_www_drain_support_one_message (system_tests_link_routes.LinkRouteTest) ... ok
test_yyy_delivery_tag (system_tests_link_routes.LinkRouteTest) ... ok
test_zzz_qdmanage_delete_link_route (system_tests_link_routes.LinkRouteTest) ... FAIL
ERROR

======================================================================
ERROR: tearDownClass (system_tests_link_routes.LinkRouteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30360 error: exit code 42, expected 0
qdrouterd -c D.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_link_routes/LinkRouteTest/setUpClass/D-4.cmd
>>>>
==30360== Memcheck, a memory error detector
==30360== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30360== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30360== Command: qdrouterd -c D.conf -I /home/aconway/dispatch/python
==30360== 
==30360== 
==30360== HEAP SUMMARY:
==30360==     in use at exit: 5,157,098 bytes in 29,291 blocks
==30360==   total heap usage: 189,328 allocs, 160,037 frees, 27,698,972 bytes allocated
==30360== 
==30360== 3,901 (2,368 direct, 1,533 indirect) bytes in 1 blocks are definitely lost in loss record 3,357 of 3,505
==30360==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30360==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30360==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30360==    by 0x4E83588: qd_server_connection (server.c:500)
==30360==    by 0x4E84F4C: on_accept (server.c:531)
==30360==    by 0x4E84F4C: handle_listener (server.c:698)
==30360==    by 0x4E84F4C: handle (server.c:842)
==30360==    by 0x4E85AAF: thread_run (server.c:919)
==30360==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30360==    by 0x6064BBE: clone (clone.S:97)
==30360== 
==30360== LEAK SUMMARY:
==30360==    definitely lost: 2,368 bytes in 1 blocks
==30360==    indirectly lost: 1,533 bytes in 21 blocks
==30360==      possibly lost: 1,343,425 bytes in 5,298 blocks
==30360==    still reachable: 3,809,772 bytes in 23,971 blocks
==30360==         suppressed: 0 bytes in 0 blocks
==30360== Reachable blocks (those to which a pointer was found) are not shown.
==30360== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30360== 
==30360== For counts of detected and suppressed errors, rerun with: -v
==30360== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

Process 30359 error: exit code 42, expected 0
qdrouterd -c C.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_link_routes/LinkRouteTest/setUpClass/C-3.cmd
>>>>
==30359== Memcheck, a memory error detector
==30359== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30359== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30359== Command: qdrouterd -c C.conf -I /home/aconway/dispatch/python
==30359== 
==30359== 
==30359== HEAP SUMMARY:
==30359==     in use at exit: 5,384,620 bytes in 30,627 blocks
==30359==   total heap usage: 258,614 allocs, 227,987 frees, 35,695,476 bytes allocated
==30359== 
==30359== 64 bytes in 1 blocks are definitely lost in loss record 1,051 of 3,557
==30359==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30359==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30359==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30359==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30359==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30359==    by 0x4E84DEC: handle (server.c:901)
==30359==    by 0x4E85AAF: thread_run (server.c:919)
==30359==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30359==    by 0x6064BBE: clone (clone.S:97)
==30359== 
==30359== 64 bytes in 1 blocks are definitely lost in loss record 1,052 of 3,557
==30359==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30359==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30359==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30359==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30359==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30359==    by 0x4E84DEC: handle (server.c:901)
==30359==    by 0x4E85AAF: thread_run (server.c:919)
==30359==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30359==    by 0x4016AB: main_process (main.c:111)
==30359==    by 0x40146E: main (main.c:318)
==30359== 
==30359== 256 bytes in 4 blocks are definitely lost in loss record 2,355 of 3,557
==30359==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30359==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30359==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30359==    by 0x4E60099: setup_outgoing_link (container.c:99)
==30359==    by 0x4E60099: qd_container_handle_event (container.c:447)
==30359==    by 0x4E84DEC: handle (server.c:901)
==30359==    by 0x4E85AAF: thread_run (server.c:919)
==30359==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30359==    by 0x6064BBE: clone (clone.S:97)
==30359== 
==30359== 57,283 (64 direct, 57,219 indirect) bytes in 1 blocks are definitely lost in loss record 3,548 of 3,557
==30359==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30359==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30359==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30359==    by 0x4E7AADA: qdr_add_link_ref (router_core.c:384)
==30359==    by 0x4E74E76: qdr_link_enqueue_work_CT (connections.c:540)
==30359==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30359==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30359==    by 0x6064BBE: clone (clone.S:97)
==30359== 
==30359== LEAK SUMMARY:
==30359==    definitely lost: 448 bytes in 7 blocks
==30359==    indirectly lost: 57,219 bytes in 382 blocks
==30359==      possibly lost: 1,433,710 bytes in 5,794 blocks
==30359==    still reachable: 3,893,243 bytes in 24,444 blocks
==30359==         suppressed: 0 bytes in 0 blocks
==30359== Reachable blocks (those to which a pointer was found) are not shown.
==30359== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30359== 
==30359== For counts of detected and suppressed errors, rerun with: -v
==30359== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 0 from 0)
<<<<

Process 30358 error: exit code 42, expected 0
qdrouterd -c B.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_link_routes/LinkRouteTest/setUpClass/B-2.cmd
>>>>
==30358== Memcheck, a memory error detector
==30358== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30358== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30358== Command: qdrouterd -c B.conf -I /home/aconway/dispatch/python
==30358== 
==30358== 
==30358== HEAP SUMMARY:
==30358==     in use at exit: 5,398,863 bytes in 30,718 blocks
==30358==   total heap usage: 285,383 allocs, 254,665 frees, 38,374,539 bytes allocated
==30358== 
==30358== 64 bytes in 1 blocks are definitely lost in loss record 1,085 of 3,606
==30358==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30358==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30358==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30358==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30358==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30358==    by 0x4E84DEC: handle (server.c:901)
==30358==    by 0x4E85AAF: thread_run (server.c:919)
==30358==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30358==    by 0x4016AB: main_process (main.c:111)
==30358==    by 0x40146E: main (main.c:318)
==30358== 
==30358== 64 bytes in 1 blocks are definitely lost in loss record 1,086 of 3,606
==30358==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30358==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30358==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30358==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30358==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30358==    by 0x4E84DEC: handle (server.c:901)
==30358==    by 0x4E85AAF: thread_run (server.c:919)
==30358==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30358==    by 0x6064BBE: clone (clone.S:97)
==30358== 
==30358== 128 bytes in 2 blocks are definitely lost in loss record 2,074 of 3,606
==30358==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30358==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30358==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30358==    by 0x4E624AE: qd_hash_internal_insert (hash.c:156)
==30358==    by 0x4E624AE: qd_hash_insert (hash.c:168)
==30358==    by 0x4E7A615: qdr_add_local_address_CT (router_core.c:309)
==30358==    by 0x4E7C7F7: qdr_route_table_setup_CT (route_tables.c:229)
==30358==    by 0x4E7AEBD: router_core_thread (router_core_thread.c:52)
==30358==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30358==    by 0x6064BBE: clone (clone.S:97)
==30358== 
==30358== 128 bytes in 2 blocks are definitely lost in loss record 2,075 of 3,606
==30358==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30358==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30358==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30358==    by 0x4E60099: setup_outgoing_link (container.c:99)
==30358==    by 0x4E60099: qd_container_handle_event (container.c:447)
==30358==    by 0x4E84DEC: handle (server.c:901)
==30358==    by 0x4E85AAF: thread_run (server.c:919)
==30358==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30358==    by 0x6064BBE: clone (clone.S:97)
==30358== 
==30358== 320 (128 direct, 192 indirect) bytes in 1 blocks are definitely lost in loss record 2,688 of 3,606
==30358==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30358==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30358==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30358==    by 0x4E78C3C: qdr_route_declare_id_CT.isra.2 (route_control.c:65)
==30358==    by 0x4E796C5: qdr_route_add_link_route_CT (route_control.c:334)
==30358==    by 0x4E73212: qdra_config_link_route_create_CT (agent_config_link_route.c:449)
==30358==    by 0x4E6F3E0: qdr_manage_create_CT (agent.c:375)
==30358==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30358==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30358==    by 0x6064BBE: clone (clone.S:97)
==30358== 
==30358== LEAK SUMMARY:
==30358==    definitely lost: 512 bytes in 7 blocks
==30358==    indirectly lost: 192 bytes in 2 blocks
==30358==      possibly lost: 1,466,274 bytes in 6,008 blocks
==30358==    still reachable: 3,931,885 bytes in 24,701 blocks
==30358==         suppressed: 0 bytes in 0 blocks
==30358== Reachable blocks (those to which a pointer was found) are not shown.
==30358== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30358== 
==30358== For counts of detected and suppressed errors, rerun with: -v
==30358== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 0 from 0)
<<<<

Process 30357 error: exit code 42, expected 0
qdrouterd -c A.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_link_routes/LinkRouteTest/setUpClass/A-1.cmd
>>>>
==30357== Memcheck, a memory error detector
==30357== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30357== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30357== Command: qdrouterd -c A.conf -I /home/aconway/dispatch/python
==30357== 
==30357== 
==30357== HEAP SUMMARY:
==30357==     in use at exit: 5,177,031 bytes in 29,420 blocks
==30357==   total heap usage: 203,181 allocs, 173,761 frees, 29,730,755 bytes allocated
==30357== 
==30357== 64 bytes in 1 blocks are definitely lost in loss record 1,079 of 3,544
==30357==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30357==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30357==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30357==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30357==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30357==    by 0x4E84DEC: handle (server.c:901)
==30357==    by 0x4E85AAF: thread_run (server.c:919)
==30357==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30357==    by 0x4016AB: main_process (main.c:111)
==30357==    by 0x40146E: main (main.c:318)
==30357== 
==30357== LEAK SUMMARY:
==30357==    definitely lost: 64 bytes in 1 blocks
==30357==    indirectly lost: 0 bytes in 0 blocks
==30357==      possibly lost: 1,363,832 bytes in 5,423 blocks
==30357==    still reachable: 3,813,135 bytes in 23,996 blocks
==30357==         suppressed: 0 bytes in 0 blocks
==30357== Reachable blocks (those to which a pointer was found) are not shown.
==30357== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30357== 
==30357== For counts of detected and suppressed errors, rerun with: -v
==30357== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

======================================================================
FAIL: test_bbb_qdstat_link_routes_routerB (system_tests_link_routes.LinkRouteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_link_routes.py", line 258, in test_bbb_qdstat_link_routes_routerB
    self.assertEqual(len(parts), 12)
AssertionError: 17 != 12

======================================================================
FAIL: test_zzz_qdmanage_delete_link_route (system_tests_link_routes.LinkRouteTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_link_routes.py", line 592, in test_zzz_qdmanage_delete_link_route
    self.assertEquals(out.rstrip(), '[]')
AssertionError: "==30563== Memcheck, a memory error detector\n==30563== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==30563== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==30563== Command: /home/aconway/dispatch/tools/qdmanage QUERY --type=linkRoute --bus amqp://0.0.0.0:29805 --indent=-1 --timeout 10.0\n==30563== \n[]" != '[]'

----------------------------------------------------------------------
Ran 21 tests in 24.614s

FAILED (failures=2, errors=1)

      Start 14: system_tests_autolinks
14/35 Test #14: system_tests_autolinks ....................***Failed    8.57 sec
test_01_autolink_attach (system_tests_autolinks.AutolinkTest) ... ok
test_02_autolink_credit (system_tests_autolinks.AutolinkTest) ... ok
test_03_autolink_sender (system_tests_autolinks.AutolinkTest) ... ok
test_04_autolink_receiver (system_tests_autolinks.AutolinkTest) ... ok
test_05_inter_container_transfer (system_tests_autolinks.AutolinkTest) ... ok
test_06_manage_autolinks (system_tests_autolinks.AutolinkTest) ... ERROR:root:Connectivity to the peer container was lost
ERROR:root:Connectivity to the peer container was lost
ERROR:root:Connectivity to the peer container was lost
ERROR:root:Connectivity to the peer container was lost
ERROR:root:Connectivity to the peer container was lost
ok
test_07_autolink_attach_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
test_08_autolink_sender_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
test_09_autolink_receiver_with_ext_addr (system_tests_autolinks.AutolinkTest) ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_autolinks.AutolinkTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30570 error: exit code 42, expected 0
qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/test-router-1.cmd
>>>>
==30570== Memcheck, a memory error detector
==30570== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30570== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30570== Command: qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
==30570== 
==30570== 
==30570== HEAP SUMMARY:
==30570==     in use at exit: 5,116,375 bytes in 29,095 blocks
==30570==   total heap usage: 347,179 allocs, 318,084 frees, 38,819,203 bytes allocated
==30570== 
==30570== 64 bytes in 1 blocks are definitely lost in loss record 1,052 of 3,467
==30570==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30570==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30570==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30570==    by 0x4E624AE: qd_hash_internal_insert (hash.c:156)
==30570==    by 0x4E624AE: qd_hash_insert (hash.c:168)
==30570==    by 0x4E7BF72: qdr_subscribe_CT (route_tables.c:671)
==30570==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30570==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30570==    by 0x6064BBE: clone (clone.S:97)
==30570== 
==30570== 1,614 (128 direct, 1,486 indirect) bytes in 1 blocks are definitely lost in loss record 3,169 of 3,467
==30570==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30570==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30570==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30570==    by 0x4E78E86: qdr_route_add_auto_link_CT (route_control.c:397)
==30570==    by 0x4E71E6D: qdra_config_auto_link_create_CT (agent_config_auto_link.c:423)
==30570==    by 0x4E6F3C8: qdr_manage_create_CT (agent.c:376)
==30570==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30570==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30570==    by 0x6064BBE: clone (clone.S:97)
==30570== 
==30570== LEAK SUMMARY:
==30570==    definitely lost: 192 bytes in 2 blocks
==30570==    indirectly lost: 1,486 bytes in 21 blocks
==30570==      possibly lost: 1,317,152 bytes in 5,195 blocks
==30570==    still reachable: 3,797,545 bytes in 23,877 blocks
==30570==         suppressed: 0 bytes in 0 blocks
==30570== Reachable blocks (those to which a pointer was found) are not shown.
==30570== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30570== 
==30570== For counts of detected and suppressed errors, rerun with: -v
==30570== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
<<<<

----------------------------------------------------------------------
Ran 9 tests in 8.527s

FAILED (errors=1)

      Start 15: system_tests_drain
15/35 Test #15: system_tests_drain ........................***Failed    5.93 sec
test_drain_support_1_all_messages (system_tests_drain.DrainSupportTest) ... ok
test_drain_support_2_one_message (system_tests_drain.DrainSupportTest) ... ok
test_drain_support_3_no_messages (system_tests_drain.DrainSupportTest) ... ok
test_drain_support_4_no_more_messages (system_tests_drain.DrainSupportTest) ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_drain.DrainSupportTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30605 error: exit code 42, expected 0
qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_drain/DrainSupportTest/setUpClass/test-router-1.cmd
>>>>
==30605== Memcheck, a memory error detector
==30605== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30605== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30605== Command: qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
==30605== 
==30605== 
==30605== HEAP SUMMARY:
==30605==     in use at exit: 5,076,923 bytes in 28,829 blocks
==30605==   total heap usage: 183,933 allocs, 155,104 frees, 26,697,410 bytes allocated
==30605== 
==30605== 128 bytes in 2 blocks are definitely lost in loss record 2,005 of 3,444
==30605==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30605==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30605==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30605==    by 0x4E60099: setup_outgoing_link (container.c:99)
==30605==    by 0x4E60099: qd_container_handle_event (container.c:447)
==30605==    by 0x4E84DEC: handle (server.c:901)
==30605==    by 0x4E85AAF: thread_run (server.c:919)
==30605==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30605==    by 0x4016AB: main_process (main.c:111)
==30605==    by 0x40146E: main (main.c:318)
==30605== 
==30605== 128 bytes in 2 blocks are definitely lost in loss record 2,006 of 3,444
==30605==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30605==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30605==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30605==    by 0x4E60099: setup_outgoing_link (container.c:99)
==30605==    by 0x4E60099: qd_container_handle_event (container.c:447)
==30605==    by 0x4E84DEC: handle (server.c:901)
==30605==    by 0x4E85AAF: thread_run (server.c:919)
==30605==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30605==    by 0x6064BBE: clone (clone.S:97)
==30605== 
==30605== LEAK SUMMARY:
==30605==    definitely lost: 256 bytes in 4 blocks
==30605==    indirectly lost: 0 bytes in 0 blocks
==30605==      possibly lost: 1,307,835 bytes in 5,158 blocks
==30605==    still reachable: 3,768,832 bytes in 23,667 blocks
==30605==         suppressed: 0 bytes in 0 blocks
==30605== Reachable blocks (those to which a pointer was found) are not shown.
==30605== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30605== 
==30605== For counts of detected and suppressed errors, rerun with: -v
==30605== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
<<<<

----------------------------------------------------------------------
Ran 4 tests in 5.885s

FAILED (errors=1)

      Start 16: system_tests_management
16/35 Test #16: system_tests_management ...................***Failed   21.95 sec
test_bad_query (system_tests_management.ManagementTest)
Test that various badly formed queries get the proper response ... ok
test_connection (system_tests_management.ManagementTest)
Verify there is at least one connection ... ok
test_create_config_address (system_tests_management.ManagementTest) ... ok
test_create_config_address_pattern (system_tests_management.ManagementTest) ... ok
test_create_listener (system_tests_management.ManagementTest)
Create a new listener on a running router ... ok
test_dummy (system_tests_management.ManagementTest)
Test all operations on the dummy test entity ... ok
test_entity_names (system_tests_management.ManagementTest) ... ok
test_get_attributes (system_tests_management.ManagementTest) ... ok
test_get_operations (system_tests_management.ManagementTest) ... ok
test_get_schema (system_tests_management.ManagementTest) ... ok
test_get_types (system_tests_management.ManagementTest) ... ok
test_link (system_tests_management.ManagementTest)
Verify we can find our own reply-to address in links ... ok
test_log (system_tests_management.ManagementTest)
Create, update and query log entities ... ok
test_query_attributes (system_tests_management.ManagementTest)
Query with attributes only ... ok
test_query_type (system_tests_management.ManagementTest)
Query with type only ... ok
test_query_type_attributes (system_tests_management.ManagementTest)
Query with type and attribute names ... ok
test_remote_node (system_tests_management.ManagementTest)
Test that we can access management info of remote nodes using get_mgmt_nodes addresses ... ok
test_router (system_tests_management.ManagementTest)
Verify router counts match entity counts ... ok
test_router_node (system_tests_management.ManagementTest)
Test node entity in a trio of linked routers ... ok
test_standalone_no_inter_router (system_tests_management.ManagementTest)
Verify that we do not allow inter-router connectors or listeners in standalone mode ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_management.ManagementTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30631 error: exit code 42, expected 0
qdrouterd -c router0.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/router0-2.cmd
>>>>
==30631== Memcheck, a memory error detector
==30631== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30631== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30631== Command: qdrouterd -c router0.conf -I /home/aconway/dispatch/python
==30631== 
==30631== 
==30631== HEAP SUMMARY:
==30631==     in use at exit: 5,309,226 bytes in 30,168 blocks
==30631==   total heap usage: 228,468 allocs, 198,300 frees, 32,594,244 bytes allocated
==30631== 
==30631== 13,908 (64 direct, 13,844 indirect) bytes in 1 blocks are definitely lost in loss record 3,493 of 3,544
==30631==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30631==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30631==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30631==    by 0x4E7AADA: qdr_add_link_ref (router_core.c:384)
==30631==    by 0x4E758CF: qdr_link_inbound_first_attach_CT (connections.c:1336)
==30631==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30631==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30631==    by 0x6064BBE: clone (clone.S:97)
==30631== 
==30631== LEAK SUMMARY:
==30631==    definitely lost: 64 bytes in 1 blocks
==30631==    indirectly lost: 13,844 bytes in 92 blocks
==30631==      possibly lost: 1,421,695 bytes in 5,755 blocks
==30631==    still reachable: 3,873,623 bytes in 24,320 blocks
==30631==         suppressed: 0 bytes in 0 blocks
==30631== Reachable blocks (those to which a pointer was found) are not shown.
==30631== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30631== 
==30631== For counts of detected and suppressed errors, rerun with: -v
==30631== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

Process 30630 error: exit code 42, expected 0
qdrouterd -c solo.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/solo-1.cmd
>>>>
==30630== Memcheck, a memory error detector
==30630== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30630== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30630== Command: qdrouterd -c solo.conf -I /home/aconway/dispatch/python
==30630== 
==30630== 
==30630== HEAP SUMMARY:
==30630==     in use at exit: 5,279,181 bytes in 30,366 blocks
==30630==   total heap usage: 330,323 allocs, 299,957 frees, 46,388,082 bytes allocated
==30630== 
==30630== 64 bytes in 1 blocks are definitely lost in loss record 1,035 of 3,495
==30630==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30630==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30630==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30630==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30630==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30630==    by 0x4E84DEC: handle (server.c:901)
==30630==    by 0x4E85AAF: thread_run (server.c:919)
==30630==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30630==    by 0x6064BBE: clone (clone.S:97)
==30630== 
==30630== 192 bytes in 3 blocks are definitely lost in loss record 2,187 of 3,495
==30630==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30630==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30630==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30630==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30630==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30630==    by 0x4E84DEC: handle (server.c:901)
==30630==    by 0x4E85AAF: thread_run (server.c:919)
==30630==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30630==    by 0x4016AB: main_process (main.c:111)
==30630==    by 0x40146E: main (main.c:318)
==30630== 
==30630== LEAK SUMMARY:
==30630==    definitely lost: 256 bytes in 4 blocks
==30630==    indirectly lost: 0 bytes in 0 blocks
==30630==      possibly lost: 1,394,321 bytes in 5,693 blocks
==30630==    still reachable: 3,884,604 bytes in 24,669 blocks
==30630==         suppressed: 0 bytes in 0 blocks
==30630== Reachable blocks (those to which a pointer was found) are not shown.
==30630== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30630== 
==30630== For counts of detected and suppressed errors, rerun with: -v
==30630== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
<<<<

----------------------------------------------------------------------
Ran 20 tests in 21.910s

FAILED (errors=1)

      Start 17: system_tests_one_router
17/35 Test #17: system_tests_one_router ...................***Failed   37.61 sec
test_01_pre_settled (system_tests_one_router.OneRouterTest) ... ok
test_02a_multicast_unsettled (system_tests_one_router.OneRouterTest) ... ok
test_02b_disp_to_closed_connection (system_tests_one_router.OneRouterTest) ... ok
test_02c_sender_settles_first (system_tests_one_router.OneRouterTest) ... ok
test_03_propagated_disposition (system_tests_one_router.OneRouterTest) ... ok
test_04_unsettled_undeliverable (system_tests_one_router.OneRouterTest) ... ok
test_05_three_ack (system_tests_one_router.OneRouterTest) ... ok
test_08_message_annotations (system_tests_one_router.OneRouterTest) ... ok
test_08a_strip_message_annotations_custom (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_both (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_both_custom (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_in (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_no (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_no_add_trace (system_tests_one_router.OneRouterTest) ... ok
test_08a_test_strip_message_annotations_out (system_tests_one_router.OneRouterTest) ... ok
test_09_management (system_tests_one_router.OneRouterTest) ... ok
test_09a_management_no_reply (system_tests_one_router.OneRouterTest) ... ok
test_09c_management_get_operations (system_tests_one_router.OneRouterTest) ... ok
test_09d_management_not_implemented (system_tests_one_router.OneRouterTest) ... ok
test_10_semantics_multicast (system_tests_one_router.OneRouterTest) ... ok
test_11_semantics_closest (system_tests_one_router.OneRouterTest) ... ok
test_12_semantics_spread (system_tests_one_router.OneRouterTest) ... ok
test_13_to_override (system_tests_one_router.OneRouterTest) ... ok
test_14_send_settle_mode_settled (system_tests_one_router.OneRouterTest) ... ok
test_15_excess_deliveries_released (system_tests_one_router.OneRouterTest) ... ok
test_16_multicast_unsettled (system_tests_one_router.OneRouterTest) ... ok
test_16a_multicast_no_receivcer (system_tests_one_router.OneRouterTest) ... ok
test_18_released_vs_modified (system_tests_one_router.OneRouterTest) ... ok
test_19_appearance_of_balance (system_tests_one_router.OneRouterTest) ... ok
test_20_batched_settlement (system_tests_one_router.OneRouterTest) ... ok
test_21_presettled_overflow (system_tests_one_router.OneRouterTest) ... ok
test_22_large_streaming_test (system_tests_one_router.OneRouterTest) ... ok
test_25_reject_coordinator (system_tests_one_router.OneRouterTest) ... FAIL
test_27_create_unavailable_sender (system_tests_one_router.OneRouterTest) ... ok
test_28_create_unavailable_receiver (system_tests_one_router.OneRouterTest) ... ok
test_connection_properties (system_tests_one_router.OneRouterTest) ... ok
test_listen_error (system_tests_one_router.OneRouterTest)
Make sure a router exits if a initial listener fails, doesn't hang ... FAIL
test_reject_disposition (system_tests_one_router.OneRouterTest) ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_one_router.OneRouterTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30723 error: exit code 42, expected 0
qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_one_router/OneRouterTest/setUpClass/test-router-1.cmd
>>>>
==30723== Memcheck, a memory error detector
==30723== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30723== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30723== Command: qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
==30723== 
==30723== 
==30723== HEAP SUMMARY:
==30723==     in use at exit: 6,243,077 bytes in 32,008 blocks
==30723==   total heap usage: 451,927 allocs, 419,919 frees, 177,638,759 bytes allocated
==30723== 
==30723== 768 bytes in 12 blocks are definitely lost in loss record 3,006 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E605A5: setup_incoming_link (container.c:134)
==30723==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==30723==    by 0x4E84DEC: handle (server.c:901)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30723==    by 0x4016AB: main_process (main.c:111)
==30723==    by 0x40146E: main (main.c:318)
==30723== 
==30723== 2,560 (128 direct, 2,432 indirect) bytes in 1 blocks are definitely lost in loss record 3,276 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E7D56B: qdr_terminus (terminus.c:44)
==30723==    by 0x4E80904: AMQP_outgoing_link_handler (router_node.c:602)
==30723==    by 0x4E84DEC: handle (server.c:901)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== 2,624 bytes in 41 blocks are definitely lost in loss record 3,278 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E60099: setup_outgoing_link (container.c:99)
==30723==    by 0x4E60099: qd_container_handle_event (container.c:447)
==30723==    by 0x4E84DEC: handle (server.c:901)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== 4,604 (2,368 direct, 2,236 indirect) bytes in 1 blocks are definitely lost in loss record 3,393 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E83588: qd_server_connection (server.c:500)
==30723==    by 0x4E84F4C: on_accept (server.c:531)
==30723==    by 0x4E84F4C: handle_listener (server.c:698)
==30723==    by 0x4E84F4C: handle (server.c:842)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== 170,624 (13,760 direct, 156,864 indirect) bytes in 43 blocks are definitely lost in loss record 3,538 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E7DE29: qdr_link_deliver (transfer.c:47)
==30723==    by 0x4E818A6: AMQP_rx_handler (router_node.c:453)
==30723==    by 0x4E6043E: do_receive (container.c:175)
==30723==    by 0x4E6043E: qd_container_handle_event (container.c:523)
==30723==    by 0x4E84DEC: handle (server.c:901)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x4E85B5F: qd_server_run (server.c:1157)
==30723==    by 0x4016AB: main_process (main.c:111)
==30723==    by 0x40146E: main (main.c:318)
==30723== 
==30723== 174,592 (14,080 direct, 160,512 indirect) bytes in 44 blocks are definitely lost in loss record 3,539 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E77098: qdr_forward_new_delivery_CT (forwarder.c:104)
==30723==    by 0x4E77ADA: qdr_forward_multicast_CT (forwarder.c:275)
==30723==    by 0x4E7EDC0: qdr_link_forward_CT (transfer.c:758)
==30723==    by 0x4E7F493: qdr_link_deliver_CT (transfer.c:912)
==30723==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== 178,560 (14,400 direct, 164,160 indirect) bytes in 45 blocks are definitely lost in loss record 3,540 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E77098: qdr_forward_new_delivery_CT (forwarder.c:104)
==30723==    by 0x4E774EB: qdr_forward_balanced_CT (forwarder.c:657)
==30723==    by 0x4E7EDC0: qdr_link_forward_CT (transfer.c:758)
==30723==    by 0x4E7F493: qdr_link_deliver_CT (transfer.c:912)
==30723==    by 0x4E7B0C3: router_core_thread (router_core_thread.c:83)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== 468,224 (37,760 direct, 430,464 indirect) bytes in 118 blocks are definitely lost in loss record 3,543 of 3,545
==30723==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30723==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30723==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30723==    by 0x4E7DE29: qdr_link_deliver (transfer.c:47)
==30723==    by 0x4E818A6: AMQP_rx_handler (router_node.c:453)
==30723==    by 0x4E6043E: do_receive (container.c:175)
==30723==    by 0x4E6043E: qd_container_handle_event (container.c:523)
==30723==    by 0x4E84DEC: handle (server.c:901)
==30723==    by 0x4E85AAF: thread_run (server.c:919)
==30723==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30723==    by 0x6064BBE: clone (clone.S:97)
==30723== 
==30723== LEAK SUMMARY:
==30723==    definitely lost: 85,888 bytes in 305 blocks
==30723==    indirectly lost: 916,668 bytes in 1,823 blocks
==30723==      possibly lost: 1,379,319 bytes in 5,663 blocks
==30723==    still reachable: 3,861,202 bytes in 24,217 blocks
==30723==         suppressed: 0 bytes in 0 blocks
==30723== Reachable blocks (those to which a pointer was found) are not shown.
==30723== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30723== 
==30723== For counts of detected and suppressed errors, rerun with: -v
==30723== ERROR SUMMARY: 8 errors from 8 contexts (suppressed: 0 from 0)
<<<<

======================================================================
FAIL: test_25_reject_coordinator (system_tests_one_router.OneRouterTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_one_router.py", line 1152, in test_25_reject_coordinator
    self.assertTrue(test.passed)
AssertionError: False is not true

======================================================================
FAIL: test_listen_error (system_tests_one_router.OneRouterTest)
Make sure a router exits if a initial listener fails, doesn't hang
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_one_router.py", line 68, in test_listen_error
    self.assertEqual(1, r.wait())
AssertionError: 1 != 42

----------------------------------------------------------------------
Ran 38 tests in 37.567s

FAILED (failures=2, errors=1)

      Start 18: system_tests_handle_failover
18/35 Test #18: system_tests_handle_failover ..............***Failed   22.32 sec
test_connector_has_failover_list (system_tests_handle_failover.FailoverTest) ... ERROR
test_remove_router_B (system_tests_handle_failover.FailoverTest) ... ERROR
ERROR

======================================================================
ERROR: test_connector_has_failover_list (system_tests_handle_failover.FailoverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_handle_failover.py", line 115, in test_connector_has_failover_list
    output = json.loads(self.run_qdmanage(query_command))
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

======================================================================
ERROR: test_remove_router_B (system_tests_handle_failover.FailoverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_handle_failover.py", line 133, in test_remove_router_B
    output = json.loads(self.run_qdmanage(query_command, address=self.routers[1].addresses[0]))
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

======================================================================
ERROR: tearDownClass (system_tests_handle_failover.FailoverTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30914 error: exit code 42, expected 0
qdrouterd -c QDR.C.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_handle_failover/FailoverTest/setUpClass/QDR.C-3.cmd
>>>>
==30914== Memcheck, a memory error detector
==30914== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30914== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30914== Command: qdrouterd -c QDR.C.conf -I /home/aconway/dispatch/python
==30914== 
==30914== 
==30914== HEAP SUMMARY:
==30914==     in use at exit: 5,165,313 bytes in 29,347 blocks
==30914==   total heap usage: 184,543 allocs, 155,196 frees, 26,403,128 bytes allocated
==30914== 
==30914== 6,111 (2,368 direct, 3,743 indirect) bytes in 1 blocks are definitely lost in loss record 3,409 of 3,512
==30914==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30914==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30914==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30914==    by 0x4E83588: qd_server_connection (server.c:500)
==30914==    by 0x4E84F4C: on_accept (server.c:531)
==30914==    by 0x4E84F4C: handle_listener (server.c:698)
==30914==    by 0x4E84F4C: handle (server.c:842)
==30914==    by 0x4E85AAF: thread_run (server.c:919)
==30914==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30914==    by 0x6064BBE: clone (clone.S:97)
==30914== 
==30914== LEAK SUMMARY:
==30914==    definitely lost: 2,368 bytes in 1 blocks
==30914==    indirectly lost: 3,743 bytes in 38 blocks
==30914==      possibly lost: 1,343,145 bytes in 5,305 blocks
==30914==    still reachable: 3,816,057 bytes in 24,003 blocks
==30914==         suppressed: 0 bytes in 0 blocks
==30914== Reachable blocks (those to which a pointer was found) are not shown.
==30914== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30914== 
==30914== For counts of detected and suppressed errors, rerun with: -v
==30914== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

Process 30894 error: exit code 42, expected 0
qdrouterd -c QDR.A.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_handle_failover/FailoverTest/setUpClass/QDR.A-2.cmd
>>>>
==30894== Memcheck, a memory error detector
==30894== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30894== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30894== Command: qdrouterd -c QDR.A.conf -I /home/aconway/dispatch/python
==30894== 
==30894== 
==30894== HEAP SUMMARY:
==30894==     in use at exit: 5,287,521 bytes in 30,091 blocks
==30894==   total heap usage: 217,689 allocs, 187,598 frees, 31,096,921 bytes allocated
==30894== 
==30894== 4,632 (2,368 direct, 2,264 indirect) bytes in 1 blocks are definitely lost in loss record 3,400 of 3,534
==30894==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30894==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30894==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30894==    by 0x4E83588: qd_server_connection (server.c:500)
==30894==    by 0x4E84F4C: on_accept (server.c:531)
==30894==    by 0x4E84F4C: handle_listener (server.c:698)
==30894==    by 0x4E84F4C: handle (server.c:842)
==30894==    by 0x4E85AAF: thread_run (server.c:919)
==30894==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30894==    by 0x6064BBE: clone (clone.S:97)
==30894== 
==30894== LEAK SUMMARY:
==30894==    definitely lost: 2,368 bytes in 1 blocks
==30894==    indirectly lost: 2,264 bytes in 30 blocks
==30894==      possibly lost: 1,424,205 bytes in 5,779 blocks
==30894==    still reachable: 3,858,684 bytes in 24,281 blocks
==30894==         suppressed: 0 bytes in 0 blocks
==30894== Reachable blocks (those to which a pointer was found) are not shown.
==30894== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30894== 
==30894== For counts of detected and suppressed errors, rerun with: -v
==30894== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

----------------------------------------------------------------------
Ran 2 tests in 22.275s

FAILED (errors=3)

      Start 19: system_tests_default_distribution
19/35 Test #19: system_tests_default_distribution .........***Failed    6.24 sec
test_anonymous_sender (system_tests_default_distribution.DefaultDistributionTest) ... ok
test_create_unavailable_receiver (system_tests_default_distribution.DefaultDistributionTest) ... ok
test_create_unavailable_sender (system_tests_default_distribution.DefaultDistributionTest) ... ok
test_general (system_tests_default_distribution.DefaultDistributionTest) ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_default_distribution.DefaultDistributionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 30963 error: exit code 42, expected 0
qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_default_distribution/DefaultDistributionTest/setUpClass/test-router-1.cmd
>>>>
==30963== Memcheck, a memory error detector
==30963== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==30963== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==30963== Command: qdrouterd -c test-router.conf -I /home/aconway/dispatch/python
==30963== 
==30963== 
==30963== HEAP SUMMARY:
==30963==     in use at exit: 5,095,353 bytes in 28,912 blocks
==30963==   total heap usage: 182,373 allocs, 153,461 frees, 26,300,110 bytes allocated
==30963== 
==30963== 5,312 (320 direct, 4,992 indirect) bytes in 1 blocks are definitely lost in loss record 3,348 of 3,464
==30963==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==30963==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==30963==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==30963==    by 0x4E7DEFD: qdr_link_deliver_to (transfer.c:71)
==30963==    by 0x4E81589: AMQP_rx_handler (router_node.c:428)
==30963==    by 0x4E6043E: do_receive (container.c:175)
==30963==    by 0x4E6043E: qd_container_handle_event (container.c:523)
==30963==    by 0x4E84DEC: handle (server.c:901)
==30963==    by 0x4E85AAF: thread_run (server.c:919)
==30963==    by 0x54EC36C: start_thread (pthread_create.c:456)
==30963==    by 0x6064BBE: clone (clone.S:97)
==30963== 
==30963== LEAK SUMMARY:
==30963==    definitely lost: 320 bytes in 1 blocks
==30963==    indirectly lost: 4,992 bytes in 14 blocks
==30963==      possibly lost: 1,310,261 bytes in 5,167 blocks
==30963==    still reachable: 3,779,780 bytes in 23,730 blocks
==30963==         suppressed: 0 bytes in 0 blocks
==30963== Reachable blocks (those to which a pointer was found) are not shown.
==30963== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==30963== 
==30963== For counts of detected and suppressed errors, rerun with: -v
==30963== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
<<<<

----------------------------------------------------------------------
Ran 4 tests in 6.195s

FAILED (errors=1)

      Start 20: system_tests_policy
20/35 Test #20: system_tests_policy .......................***Failed   18.67 sec
test_verify_maximum_connections (system_tests_policy.AbsoluteConnectionCountLimit) ... ok
test_repeated_create_delete (system_tests_policy.LoadPolicyFromFolder) ... ERROR
test_verify_policies_are_loaded (system_tests_policy.LoadPolicyFromFolder) ... ERROR
test_verify_policy_add_update_delete (system_tests_policy.LoadPolicyFromFolder) ... ERROR
test_verify_n_receivers (system_tests_policy.SenderReceiverLimits) ... ok
test_verify_n_senders (system_tests_policy.SenderReceiverLimits) ... ok

======================================================================
ERROR: test_repeated_create_delete (system_tests_policy.LoadPolicyFromFolder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_policy.py", line 227, in test_repeated_create_delete
    rulesets = json.loads(self.run_qdmanage('query --type=vhost'))
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

======================================================================
ERROR: test_verify_policies_are_loaded (system_tests_policy.LoadPolicyFromFolder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_policy.py", line 124, in test_verify_policies_are_loaded
    rulesets = json.loads(self.run_qdmanage('query --type=vhost'))
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

======================================================================
ERROR: test_verify_policy_add_update_delete (system_tests_policy.LoadPolicyFromFolder)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_tests_policy.py", line 183, in test_verify_policy_add_update_delete
    rulesets = json.loads(self.run_qdmanage('query --type=vhost'))
  File "/usr/lib64/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

----------------------------------------------------------------------
Ran 6 tests in 18.626s

FAILED (errors=3)

      Start 21: system_tests_protocol_family
21/35 Test #21: system_tests_protocol_family ..............***Failed   18.98 sec
test_simple_pre_settled (system_tests_protocol_family.ProtocolFamilyTest) ... ok
ERROR

======================================================================
ERROR: tearDownClass (system_tests_protocol_family.ProtocolFamilyTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aconway/dispatch/tests/system_test.py", line 606, in tearDownClass
    cls.tester.teardown()
  File "/home/aconway/dispatch/tests/system_test.py", line 544, in teardown
    raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e) for e in errors]))
RuntimeError: Errors during teardown: 

Process 31068 error: exit code 42, expected 0
qdrouterd -c A.conf -I /home/aconway/dispatch/python
/home/aconway/dispatch/reldbg/tests/system_test.dir/system_tests_protocol_family/ProtocolFamilyTest/setUpClass/A-1.cmd
>>>>
==31068== Memcheck, a memory error detector
==31068== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==31068== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==31068== Command: qdrouterd -c A.conf -I /home/aconway/dispatch/python
==31068== 
==31068== 
==31068== HEAP SUMMARY:
==31068==     in use at exit: 5,323,005 bytes in 30,225 blocks
==31068==   total heap usage: 229,454 allocs, 199,229 frees, 33,338,922 bytes allocated
==31068== 
==31068== 128 bytes in 2 blocks are definitely lost in loss record 2,038 of 3,532
==31068==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==31068==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==31068==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==31068==    by 0x4E605A5: setup_incoming_link (container.c:134)
==31068==    by 0x4E605A5: qd_container_handle_event (container.c:455)
==31068==    by 0x4E84DEC: handle (server.c:901)
==31068==    by 0x4E85AAF: thread_run (server.c:919)
==31068==    by 0x54EC36C: start_thread (pthread_create.c:456)
==31068==    by 0x6064BBE: clone (clone.S:97)
==31068== 
==31068== 4,632 (2,368 direct, 2,264 indirect) bytes in 1 blocks are definitely lost in loss record 3,397 of 3,532
==31068==    at 0x4C30D47: memalign (vg_replace_malloc.c:857)
==31068==    by 0x4C30E45: posix_memalign (vg_replace_malloc.c:1020)
==31068==    by 0x4E868D8: qd_alloc (alloc_pool.c:182)
==31068==    by 0x4E83588: qd_server_connection (server.c:500)
==31068==    by 0x4E84F4C: on_accept (server.c:531)
==31068==    by 0x4E84F4C: handle_listener (server.c:698)
==31068==    by 0x4E84F4C: handle (server.c:842)
==31068==    by 0x4E85AAF: thread_run (server.c:919)
==31068==    by 0x54EC36C: start_thread (pthread_create.c:456)
==31068==    by 0x6064BBE: clone (clone.S:97)
==31068== 
==31068== LEAK SUMMARY:
==31068==    definitely lost: 2,496 bytes in 3 blocks
==31068==    indirectly lost: 2,264 bytes in 30 blocks
==31068==      possibly lost: 1,424,298 bytes in 5,765 blocks
==31068==    still reachable: 3,893,947 bytes in 24,427 blocks
==31068==         suppressed: 0 bytes in 0 blocks
==31068== Reachable blocks (those to which a pointer was found) are not shown.
==31068== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==31068== 
==31068== For counts of detected and suppressed errors, rerun with: -v
==31068== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
<<<<

{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org