You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "Jiri Daněk (Jira)" <ji...@apache.org> on 2020/04/08 15:35:00 UTC
[jira] [Commented] (PROTON-2111) python: memory leak on Container,
SSL, and SSLDomain objects
[ https://issues.apache.org/jira/browse/PROTON-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17078390#comment-17078390 ]
Jiri Daněk commented on PROTON-2111:
------------------------------------
I am thinking that this is a duplicate of PROTON-2116. I tried the reproducer (using both objgraph and gc.collect(); print(len(gc.get_objects())) and I observed that there is object count growth even in proton 0.27, but it is nowhere as dramatic as in 0.28. Then I saw the issue is fixed in 0.30, and it is also fixed in current master.
The test uses ssl. I think it is a good idea to have object leak test that uses ssl. I'll clean up the test, decide whether to keep objgraph, use guppy (that is what I have some experience with) or just the gc.get_objects, and send PR. For that reason I am not closing this issue as duplicate just yet.
{noformat}
for v in 0.27.1 0.28.0 0.29.0 0.30.0; do :(
echo $v
echo xxxxxxxxxxxxxxxxxxxxxx
TESTDIR=qpid-proton/python/tests/integration
INCLUDEDIR=qpid-proton/tests/py
PYTHONPATH=qpid-proton_${v}/build/install/lib64/proton/bindings/python:${INCLUDEDIR} \
LD_LIBRARY_PATH=qpid-proton_${v}/build/install/lib64 \
python -m unittest discover -v -s $TESTDIR -p '*_test.py'
done
0.27.1
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests) ... /home/jdanek/repos/qpid/qpid-proton/python/tests/integration/blocking_connection_fd_leak_test.py:148: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::e092:8ad:e57f:0', 39407, 0, 4294967295)>
urls = "amqps://{0}:{1}".format(*broker.get_acceptor_sockname())
ResourceWarning: Enable tracemalloc to get the object allocation traceback
amqps://:::39407
function 8365 +8365
dict 4234 +4234
tuple 3157 +3157
list 2154 +2154
weakref 1927 +1927
builtin_function_or_method 1669 +1669
member_descriptor 1564 +1564
type 1289 +1289
wrapper_descriptor 1284 +1284
getset_descriptor 1230 +1230
len gc get objects 32694
len gc get objects 32711
len gc get objects 32728
len gc get objects 32745
len gc get objects 32762
==============================================================================
Type Old_ids Current_ids New_ids Count_Deltas
==============================================================================
function 0 8365 +8365 +8365
dict 0 4258 +4258 +4258
tuple 0 3151 +3151 +3151
list 0 2163 +2163 +2163
weakref 0 1925 +1925 +1925
builtin_function_or_method 0 1667 +1667 +1667
member_descriptor 0 1564 +1564 +1564
type 0 1289 +1289 +1289
wrapper_descriptor 0 1284 +1284 +1284
getset_descriptor 0 1230 +1230 +1230
==============================================================================
[<proton._transport.SSLDomain object at 0x7fe59f19e550>, <proton._transport.SSLDomain object at 0x7fe59f19e7d0>]
[]
[<proton._transport.SSL object at 0x7fe59dbc5250>, <proton._transport.SSL object at 0x7fe59dbc5390>, <proton._transport.SSL object at 0x7fe59dbc5550>, <proton._transport.SSL object at 0x7fe59dbc5710>, <proton._transport.SSL object at 0x7fe59dbc58d0>]
ok
----------------------------------------------------------------------
Ran 1 test in 3.262s
OK
0.28.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests) ... amqps://127.0.0.1:37773
function 8423 +8423
dict 4250 +4250
tuple 3178 +3178
list 2158 +2158
weakref 1935 +1935
builtin_function_or_method 1562 +1562
member_descriptor 1553 +1553
type 1299 +1299
wrapper_descriptor 1284 +1284
getset_descriptor 1233 +1233
len gc get objects 32728
len gc get objects 32753
len gc get objects 32776
len gc get objects 32797
len gc get objects 32819
==============================================================================
Type Old_ids Current_ids New_ids Count_Deltas
==============================================================================
function 0 8423 +8423 +8423
dict 0 4290 +4290 +4290
tuple 0 3178 +3178 +3178
list 0 2166 +2166 +2166
weakref 0 1933 +1933 +1933
builtin_function_or_method 0 1560 +1560 +1560
member_descriptor 0 1553 +1553 +1553
type 0 1299 +1299 +1299
wrapper_descriptor 0 1284 +1284 +1284
getset_descriptor 0 1233 +1233 +1233
==============================================================================
[<proton._transport.SSLDomain object at 0x7fb9a1069550>, <proton._transport.SSLDomain object at 0x7fb9a10697d0>, <proton._transport.SSLDomain object at 0x7fb99f266790>, <proton._transport.SSLDomain object at 0x7fb99f266ad0>, <proton._transport.SSLDomain object at 0x7fb99f266350>, <proton._transport.SSLDomain object at 0x7fb99f266cd0>, <proton._transport.SSLDomain object at 0x7fb99f266410>, <proton._transport.SSLDomain object at 0x7fb99f266c90>, <proton._transport.SSLDomain object at 0x7fb99f266890>, <proton._transport.SSLDomain object at 0x7fb99f2669d0>, <proton._transport.SSLDomain object at 0x7fb99f266bd0>, <proton._transport.SSLDomain object at 0x7fb99f275890>, <proton._transport.SSLDomain object at 0x7fb99fab6c10>, <proton._transport.SSLDomain object at 0x7fb99fab69d0>]
[<proton._reactor.Container object at 0x7fb99facaf50>, <proton._reactor.Container object at 0x7fb99f266a10>, <proton._reactor.Container object at 0x7fb99f266d50>, <proton._reactor.Container object at 0x7fb99facae90>, <proton._reactor.Container object at 0x7fb99fab7f10>, <proton._reactor.Container object at 0x7fb99faba0d0>]
[<proton._transport.SSL object at 0x7fb99f266c10>, <proton._transport.SSL object at 0x7fb99f2750d0>, <proton._transport.SSL object at 0x7fb99f266d90>, <proton._transport.SSL object at 0x7fb99f2752d0>, <proton._transport.SSL object at 0x7fb99f275210>, <proton._transport.SSL object at 0x7fb99f275510>, <proton._transport.SSL object at 0x7fb99f275490>, <proton._transport.SSL object at 0x7fb99f2757d0>, <proton._transport.SSL object at 0x7fb99f275610>, <proton._transport.SSL object at 0x7fb99f275810>]
ok
----------------------------------------------------------------------
Ran 1 test in 0.600s
OK
0.29.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests) ... amqps://127.0.0.1:39825
function 8423 +8423
dict 4250 +4250
tuple 3179 +3179
list 2158 +2158
weakref 1935 +1935
builtin_function_or_method 1562 +1562
member_descriptor 1553 +1553
type 1299 +1299
wrapper_descriptor 1284 +1284
getset_descriptor 1233 +1233
len gc get objects 32721
len gc get objects 32746
len gc get objects 32778
len gc get objects 32798
len gc get objects 32826
==============================================================================
Type Old_ids Current_ids New_ids Count_Deltas
==============================================================================
function 0 8423 +8423 +8423
dict 0 4290 +4290 +4290
tuple 0 3178 +3178 +3178
list 0 2166 +2166 +2166
weakref 0 1933 +1933 +1933
builtin_function_or_method 0 1560 +1560 +1560
member_descriptor 0 1553 +1553 +1553
type 0 1299 +1299 +1299
wrapper_descriptor 0 1284 +1284 +1284
getset_descriptor 0 1233 +1233 +1233
==============================================================================
[<proton._transport.SSLDomain object at 0x7f4278a4a550>, <proton._transport.SSLDomain object at 0x7f4278a4a7d0>, <proton._transport.SSLDomain object at 0x7f4276c4cad0>, <proton._transport.SSLDomain object at 0x7f4276c4cc50>, <proton._transport.SSLDomain object at 0x7f4276c4cb50>, <proton._transport.SSLDomain object at 0x7f4276c4ca50>, <proton._transport.SSLDomain object at 0x7f4276c4c610>, <proton._transport.SSLDomain object at 0x7f4276c4cc10>, <proton._transport.SSLDomain object at 0x7f4276c4cd50>, <proton._transport.SSLDomain object at 0x7f4277fcf990>, <proton._transport.SSLDomain object at 0x7f4276c4c450>, <proton._transport.SSLDomain object at 0x7f4276c54950>, <proton._transport.SSLDomain object at 0x7f427749c310>, <proton._transport.SSLDomain object at 0x7f427749cb50>]
[<proton._reactor.Container object at 0x7f4276c4c950>, <proton._reactor.Container object at 0x7f42774b3e90>, <proton._reactor.Container object at 0x7f427749c990>, <proton._reactor.Container object at 0x7f427749c890>, <proton._reactor.Container object at 0x7f42774b3110>, <proton._reactor.Container object at 0x7f427749eed0>]
[<proton._transport.SSL object at 0x7f4276c540d0>, <proton._transport.SSL object at 0x7f4276c54250>, <proton._transport.SSL object at 0x7f4276c54090>, <proton._transport.SSL object at 0x7f4276c54410>, <proton._transport.SSL object at 0x7f4276c544d0>, <proton._transport.SSL object at 0x7f4276c4ca10>, <proton._transport.SSL object at 0x7f4276c54650>, <proton._transport.SSL object at 0x7f4276c549d0>, <proton._transport.SSL object at 0x7f4276c54a10>, <proton._transport.SSL object at 0x7f4276c54c90>]
ok
----------------------------------------------------------------------
Ran 1 test in 0.475s
OK
0.30.0
xxxxxxxxxxxxxxxxxxxxxx
test_send_message_ssl_no_object_leaks (blocking_connection_fd_leak_test.BlockingConnectionFDLeakTests) ... amqps://127.0.0.1:33689
function 8451 +8451
dict 4255 +4255
tuple 3200 +3200
list 2158 +2158
weakref 1939 +1939
builtin_function_or_method 1563 +1563
member_descriptor 1553 +1553
type 1303 +1303
wrapper_descriptor 1284 +1284
getset_descriptor 1237 +1237
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
len gc get objects 32773
==============================================================================
Type Old_ids Current_ids New_ids Count_Deltas
==============================================================================
function 0 8451 +8451 +8451
dict 0 4245 +4245 +4245
tuple 0 3198 +3198 +3198
list 0 2156 +2156 +2156
weakref 0 1937 +1937 +1937
builtin_function_or_method 0 1561 +1561 +1561
member_descriptor 0 1553 +1553 +1553
type 0 1303 +1303 +1303
wrapper_descriptor 0 1284 +1284 +1284
getset_descriptor 0 1237 +1237 +1237
==============================================================================
[<proton._transport.SSLDomain object at 0x7fc9f7658550>, <proton._transport.SSLDomain object at 0x7fc9f76587d0>, <proton._transport.SSLDomain object at 0x7fc9f60bf850>, <proton._transport.SSLDomain object at 0x7fc9f60bf610>]
[<proton._reactor.Container object at 0x7fc9f60bcf50>]
[]
ok
----------------------------------------------------------------------
Ran 1 test in 3.715s
OK
{noformat}
> python: memory leak on Container, SSL, and SSLDomain objects
> ------------------------------------------------------------
>
> Key: PROTON-2111
> URL: https://issues.apache.org/jira/browse/PROTON-2111
> Project: Qpid Proton
> Issue Type: Bug
> Components: python-binding
> Affects Versions: proton-c-0.28.0
> Reporter: Chenxiong Qi
> Assignee: Clifford Jansen
> Priority: Major
>
> I have an application using qpid.proton to interact with ActiveMQ broker to publish messages. Following is a fake script showing an example of the app works to send a message.
> {noformat}
> class SampleSender(proton.handlers.MessagingHandler):
> def __init__(self, msg_id, *args, **kwargs):
> super(SampleSender, self).__init__(*args, **kwargs)
> self.msg_id = msg_id
> def on_start(self, event):
> ssl_domain = proton.SSLDomain(proton.SSLDomain.MODE_CLIENT)
> ssl_domain.set_credentials(producer_config['certificate'],
> producer_config['private_key'],
> None)
> ssl_domain.set_trusted_ca_db(producer_config['trusted_certificates'])
> ssl_domain.set_peer_authentication(proton.SSLDomain.VERIFY_PEER)
> conn = event.container.connect(urls=producer_config['urls'],
> reconnect=False,
> ssl_domain=ssl_domain)
> event.container.create_sender(conn, target='topic://VirtualTopic.event')
> def on_sendable(self, event):
> msg = proton.Message(body={'msg-id': self.msg_id, 'name': 'python'})
> event.sender.send(msg)
> event.sender.close()
> event.connection.close()
> def send_msg(msg_id):
> container = proton.reactor.Container(SampleSender(msg_id))
> container.run()
> objgraph.show_growth()
> for i in range(3):
> send_msg(i + 1)
> new_ids = objgraph.get_new_ids()
> print(objgraph.at_addrs(new_ids['SSLDomain']))
> print(objgraph.at_addrs(new_ids['Container']))
> print(objgraph.at_addrs(new_ids['SSL']))
> {noformat}
> Each time to publish a message, a new Container object is created and a new handler object is created and passed to that container. After the code runs several times, Container, SSL and SSLDomain objects remain in memory and are not collected. I use objgraph to find out those objects.
> I'm using Fedora 30 and python-qpid-proton 0.28.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org