You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by ma...@datenwort.at.INVALID on 2023/07/06 13:19:38 UTC
Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75
Hello!
I guess I found a memory leak which I am not quite sure how to reproduce yet.
Our application is using WebSockets and sometimes it seems that Tomcat 9.0.75 does not always remove the „UpgradeProcessorInternal“ process from the list of „waitingProcessors“ in „org.apache.coyote.AbstractProtocol“.
I managed to create a dump (in that e-mail below) of that data from a live VM and found that some of those UpgradeProcessorInternal instances are referencing a "Closed Socket", which somehow feels odd to me. This list slowly grows. The entries are never removed.
Those UpgradeProcessorInternal-s referencing a live socket are matching exactly the number of live WebSocket sessions.
What do you think?
Env:
Tomcat: 9.0.75.0
Spring Boot: 2.7.12
Java: 17.0.7
Best regards,
Mario
connector :Connector[HTTP/1.1-2888]
waitingProcessors: 325
processor#1: UpgradeProcessorInternal@1087825622 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#2: UpgradeProcessorInternal@92090237 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#3: UpgradeProcessorInternal@1641815002 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#4: UpgradeProcessorInternal@1749164616 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#5: UpgradeProcessorInternal@508142663 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#6: UpgradeProcessorInternal@1482593872 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#7: UpgradeProcessorInternal@1176145940 socketWrapper state: Closed NioChannel msgSent=4389 msgReceived=205
processor#8: UpgradeProcessorInternal@1420482234 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#9: UpgradeProcessorInternal@31846900 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#10: UpgradeProcessorInternal@1726295282 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#11: UpgradeProcessorInternal@288536869 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#12: UpgradeProcessorInternal@969872343 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#13: UpgradeProcessorInternal@420227862 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#14: UpgradeProcessorInternal@552002299 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#15: UpgradeProcessorInternal@255507237 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#16: UpgradeProcessorInternal@724602640 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#17: UpgradeProcessorInternal@1706491794 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#18: UpgradeProcessorInternal@1925947118 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#19: UpgradeProcessorInternal@2044888516 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#20: UpgradeProcessorInternal@905596380 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#21: UpgradeProcessorInternal@897538387 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@61aa8978:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:40566] msgSent=7277 msgReceived=447
processor#22: UpgradeProcessorInternal@1070005238 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#23: UpgradeProcessorInternal@34919463 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#24: UpgradeProcessorInternal@565924232 socketWrapper state: Closed NioChannel msgSent=12 msgReceived=14
processor#25: UpgradeProcessorInternal@1146279009 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#26: UpgradeProcessorInternal@1467208007 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#27: UpgradeProcessorInternal@348911347 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#28: UpgradeProcessorInternal@38694518 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#29: UpgradeProcessorInternal@1164156250 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#30: UpgradeProcessorInternal@1675936734 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#31: UpgradeProcessorInternal@441511021 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#32: UpgradeProcessorInternal@59206586 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@5910f8ae:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50444] msgSent=3684 msgReceived=449
processor#33: UpgradeProcessorInternal@1027164423 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#34: UpgradeProcessorInternal@2101191939 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#35: UpgradeProcessorInternal@1598120319 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#36: UpgradeProcessorInternal@1601686839 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#37: UpgradeProcessorInternal@2131114820 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#38: UpgradeProcessorInternal@1695970133 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#39: UpgradeProcessorInternal@1088759468 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@626eafa5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58972] msgSent=727 msgReceived=868
processor#40: UpgradeProcessorInternal@332728223 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#41: UpgradeProcessorInternal@221860722 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#42: UpgradeProcessorInternal@773355092 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#43: UpgradeProcessorInternal@826604304 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#44: UpgradeProcessorInternal@267595687 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6c19b8c1:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50565] msgSent=2968 msgReceived=387
processor#45: UpgradeProcessorInternal@417381557 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#46: UpgradeProcessorInternal@1626896557 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#47: UpgradeProcessorInternal@1326188889 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#48: UpgradeProcessorInternal@1620503747 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#49: UpgradeProcessorInternal@1588637927 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1ce5fc5a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:33528] msgSent=5765 msgReceived=442
processor#50: UpgradeProcessorInternal@148940473 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#51: UpgradeProcessorInternal@915453133 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#52: UpgradeProcessorInternal@369115230 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#53: UpgradeProcessorInternal@405977197 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#54: UpgradeProcessorInternal@47138960 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#55: UpgradeProcessorInternal@178012412 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@13f91fab:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48880] msgSent=5 msgReceived=11
processor#56: UpgradeProcessorInternal@360143126 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#57: UpgradeProcessorInternal@1196637488 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#58: UpgradeProcessorInternal@1741581739 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2815c011:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59371] msgSent=8604 msgReceived=853
processor#59: UpgradeProcessorInternal@1053645483 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#60: UpgradeProcessorInternal@490296158 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#61: UpgradeProcessorInternal@280948950 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#62: UpgradeProcessorInternal@1763242354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#63: UpgradeProcessorInternal@170476101 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#64: UpgradeProcessorInternal@791654722 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#65: UpgradeProcessorInternal@808226370 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#66: UpgradeProcessorInternal@325893404 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#67: UpgradeProcessorInternal@1205002145 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#68: UpgradeProcessorInternal@105139248 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#69: UpgradeProcessorInternal@835317180 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#70: UpgradeProcessorInternal@1110689861 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2e9114a0:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57172] msgSent=31182 msgReceived=737
processor#71: UpgradeProcessorInternal@694149416 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#72: UpgradeProcessorInternal@1917387827 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#73: UpgradeProcessorInternal@1669990384 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#74: UpgradeProcessorInternal@482149574 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#75: UpgradeProcessorInternal@1630942539 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#76: UpgradeProcessorInternal@661587730 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#77: UpgradeProcessorInternal@334804362 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#78: UpgradeProcessorInternal@1847955546 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@f2be4c4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60237] msgSent=537 msgReceived=557
processor#79: UpgradeProcessorInternal@961085898 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#80: UpgradeProcessorInternal@1015170564 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#81: UpgradeProcessorInternal@1472660801 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#82: UpgradeProcessorInternal@1662742940 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#83: UpgradeProcessorInternal@783736881 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#84: UpgradeProcessorInternal@1033748247 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#85: UpgradeProcessorInternal@1631720904 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#86: UpgradeProcessorInternal@377835540 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
processor#87: UpgradeProcessorInternal@208792804 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#88: UpgradeProcessorInternal@27480885 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#89: UpgradeProcessorInternal@6538996 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#90: UpgradeProcessorInternal@2030787475 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#91: UpgradeProcessorInternal@1780923582 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#92: UpgradeProcessorInternal@30596938 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#93: UpgradeProcessorInternal@165266752 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#94: UpgradeProcessorInternal@1937101292 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#95: UpgradeProcessorInternal@854833769 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#96: UpgradeProcessorInternal@1653312022 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#97: UpgradeProcessorInternal@946062077 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#98: UpgradeProcessorInternal@111135744 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#99: UpgradeProcessorInternal@1407359357 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@50247a32:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56550] msgSent=17041 msgReceived=1054
processor#100: UpgradeProcessorInternal@1688084542 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#101: UpgradeProcessorInternal@1506439018 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#102: UpgradeProcessorInternal@515959904 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#103: UpgradeProcessorInternal@259281365 socketWrapper state: Closed NioChannel msgSent=12 msgReceived=12
processor#104: UpgradeProcessorInternal@364095762 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#105: UpgradeProcessorInternal@307308786 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#106: UpgradeProcessorInternal@1677683546 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#107: UpgradeProcessorInternal@318642779 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#108: UpgradeProcessorInternal@914880558 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#109: UpgradeProcessorInternal@291835852 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#110: UpgradeProcessorInternal@168338082 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#111: UpgradeProcessorInternal@1174282578 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#112: UpgradeProcessorInternal@331106577 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#113: UpgradeProcessorInternal@1617882333 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#114: UpgradeProcessorInternal@542714593 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#115: UpgradeProcessorInternal@1156269649 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#116: UpgradeProcessorInternal@201406651 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#117: UpgradeProcessorInternal@2058862092 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#118: UpgradeProcessorInternal@857502172 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#119: UpgradeProcessorInternal@996422055 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#120: UpgradeProcessorInternal@1815696637 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#121: UpgradeProcessorInternal@736438049 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@7b35f1fe:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58899] msgSent=799 msgReceived=935
processor#122: UpgradeProcessorInternal@477398194 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#123: UpgradeProcessorInternal@1219089519 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#124: UpgradeProcessorInternal@1272242963 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#125: UpgradeProcessorInternal@121254386 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#126: UpgradeProcessorInternal@1669364040 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#127: UpgradeProcessorInternal@275174062 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#128: UpgradeProcessorInternal@554430912 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#129: UpgradeProcessorInternal@1187725828 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#130: UpgradeProcessorInternal@1531691908 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#131: UpgradeProcessorInternal@446727798 socketWrapper state: Closed NioChannel msgSent=283 msgReceived=259
processor#132: UpgradeProcessorInternal@362658123 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#133: UpgradeProcessorInternal@330299253 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#134: UpgradeProcessorInternal@1417579684 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#135: UpgradeProcessorInternal@79366241 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#136: UpgradeProcessorInternal@1794180138 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#137: UpgradeProcessorInternal@773525189 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#138: UpgradeProcessorInternal@88744340 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#139: UpgradeProcessorInternal@709557909 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#140: UpgradeProcessorInternal@263285102 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#141: UpgradeProcessorInternal@1223181320 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#142: UpgradeProcessorInternal@1288060966 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@106e2540:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:61373] msgSent=9780 msgReceived=297
processor#143: UpgradeProcessorInternal@102971590 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#144: UpgradeProcessorInternal@808226506 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#145: UpgradeProcessorInternal@1228942248 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#146: UpgradeProcessorInternal@1214228151 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#147: UpgradeProcessorInternal@550718522 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#148: UpgradeProcessorInternal@546382458 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#149: UpgradeProcessorInternal@430113102 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#150: UpgradeProcessorInternal@1801235373 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#151: UpgradeProcessorInternal@613091448 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#152: UpgradeProcessorInternal@567610662 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@7d797520:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:32782] msgSent=7435 msgReceived=471
processor#153: UpgradeProcessorInternal@864215926 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#154: UpgradeProcessorInternal@257132453 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#155: UpgradeProcessorInternal@1679461613 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#156: UpgradeProcessorInternal@1792036392 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#157: UpgradeProcessorInternal@1237165381 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#158: UpgradeProcessorInternal@2095009319 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#159: UpgradeProcessorInternal@405187807 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#160: UpgradeProcessorInternal@1137415986 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#161: UpgradeProcessorInternal@1259367401 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#162: UpgradeProcessorInternal@590114774 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#163: UpgradeProcessorInternal@1979332358 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#164: UpgradeProcessorInternal@1910582044 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#165: UpgradeProcessorInternal@1205302056 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#166: UpgradeProcessorInternal@1454816183 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#167: UpgradeProcessorInternal@1127766585 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#168: UpgradeProcessorInternal@1563629106 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2cdc91f6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59195] msgSent=832 msgReceived=865
processor#169: UpgradeProcessorInternal@27650727 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#170: UpgradeProcessorInternal@1847115547 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#171: UpgradeProcessorInternal@1939276436 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#172: UpgradeProcessorInternal@2038944386 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#173: UpgradeProcessorInternal@1155354584 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#174: UpgradeProcessorInternal@97303750 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#175: UpgradeProcessorInternal@1608676584 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#176: UpgradeProcessorInternal@886887382 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#177: UpgradeProcessorInternal@620014078 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#178: UpgradeProcessorInternal@551004124 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#179: UpgradeProcessorInternal@2110011592 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#180: UpgradeProcessorInternal@1908346035 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#181: UpgradeProcessorInternal@378148743 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#182: UpgradeProcessorInternal@1156173823 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#183: UpgradeProcessorInternal@64974024 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#184: UpgradeProcessorInternal@1009466172 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#185: UpgradeProcessorInternal@536984857 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#186: UpgradeProcessorInternal@383186893 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#187: UpgradeProcessorInternal@836747459 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#188: UpgradeProcessorInternal@1077816609 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#189: UpgradeProcessorInternal@645247338 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=10
processor#190: UpgradeProcessorInternal@921863122 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#191: UpgradeProcessorInternal@1763449404 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#192: UpgradeProcessorInternal@464409230 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@519deb3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:36638] msgSent=5311 msgReceived=430
processor#193: UpgradeProcessorInternal@1278303507 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#194: UpgradeProcessorInternal@1244687124 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#195: UpgradeProcessorInternal@1930251816 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#196: UpgradeProcessorInternal@43494834 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@29f70e25:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:51288] msgSent=7 msgReceived=9
processor#197: UpgradeProcessorInternal@1612242224 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#198: UpgradeProcessorInternal@1831479299 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#199: UpgradeProcessorInternal@287185460 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#200: UpgradeProcessorInternal@1947436860 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#201: UpgradeProcessorInternal@171168002 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#202: UpgradeProcessorInternal@1436058281 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#203: UpgradeProcessorInternal@1647228700 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#204: UpgradeProcessorInternal@658189327 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#205: UpgradeProcessorInternal@1207405250 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#206: UpgradeProcessorInternal@1206888150 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#207: UpgradeProcessorInternal@11407255 socketWrapper state: Closed NioChannel msgSent=512 msgReceived=549
processor#208: UpgradeProcessorInternal@999984803 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6887705e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59146] msgSent=719 msgReceived=796
processor#209: UpgradeProcessorInternal@2079580872 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#210: UpgradeProcessorInternal@26762917 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#211: UpgradeProcessorInternal@653767625 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#212: UpgradeProcessorInternal@685357542 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#213: UpgradeProcessorInternal@2105023035 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#214: UpgradeProcessorInternal@906125638 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#215: UpgradeProcessorInternal@1739573994 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#216: UpgradeProcessorInternal@500242071 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#217: UpgradeProcessorInternal@1067985634 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#218: UpgradeProcessorInternal@1526977615 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@113386a5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56957] msgSent=11502 msgReceived=275
processor#219: UpgradeProcessorInternal@1564249713 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#220: UpgradeProcessorInternal@1271579268 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#221: UpgradeProcessorInternal@1251932623 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#222: UpgradeProcessorInternal@336201051 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#223: UpgradeProcessorInternal@1969809982 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#224: UpgradeProcessorInternal@1001859809 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#225: UpgradeProcessorInternal@1073172128 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#226: UpgradeProcessorInternal@787383222 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#227: UpgradeProcessorInternal@599435490 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#228: UpgradeProcessorInternal@202250581 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#229: UpgradeProcessorInternal@2120255293 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#230: UpgradeProcessorInternal@1888709582 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#231: UpgradeProcessorInternal@765967098 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#232: UpgradeProcessorInternal@1517419823 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#233: UpgradeProcessorInternal@1701040641 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#234: UpgradeProcessorInternal@1146615605 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#235: UpgradeProcessorInternal@117512293 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#236: UpgradeProcessorInternal@2012342422 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#237: UpgradeProcessorInternal@1973886667 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#238: UpgradeProcessorInternal@1105514123 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#239: UpgradeProcessorInternal@1632774205 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2e335b3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60332] msgSent=1124 msgReceived=510
processor#240: UpgradeProcessorInternal@1643707528 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#241: UpgradeProcessorInternal@1788721645 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#242: UpgradeProcessorInternal@1338930879 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#243: UpgradeProcessorInternal@1437970118 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#244: UpgradeProcessorInternal@825036894 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#245: UpgradeProcessorInternal@1411771222 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#246: UpgradeProcessorInternal@1641563307 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#247: UpgradeProcessorInternal@1910767760 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#248: UpgradeProcessorInternal@1376679291 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#249: UpgradeProcessorInternal@1369067244 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#250: UpgradeProcessorInternal@640417107 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@12ac4472:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56706] msgSent=725 msgReceived=838
processor#251: UpgradeProcessorInternal@1723934650 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#252: UpgradeProcessorInternal@390093373 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#253: UpgradeProcessorInternal@1679102358 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#254: UpgradeProcessorInternal@1943695966 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#255: UpgradeProcessorInternal@1675844197 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@46028bc6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59060] msgSent=1799 msgReceived=1047
processor#256: UpgradeProcessorInternal@1597795515 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#257: UpgradeProcessorInternal@449048399 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#258: UpgradeProcessorInternal@1138357335 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#259: UpgradeProcessorInternal@1150701318 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#260: UpgradeProcessorInternal@759289043 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#261: UpgradeProcessorInternal@663090198 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#262: UpgradeProcessorInternal@1804724738 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#263: UpgradeProcessorInternal@1681171367 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#264: UpgradeProcessorInternal@894140638 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#265: UpgradeProcessorInternal@1268650502 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#266: UpgradeProcessorInternal@201391001 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
processor#267: UpgradeProcessorInternal@4454872 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@deddb3c:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50534] msgSent=2955 msgReceived=422
processor#268: UpgradeProcessorInternal@1292060318 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#269: UpgradeProcessorInternal@1799841497 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#270: UpgradeProcessorInternal@1293709 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
processor#271: UpgradeProcessorInternal@1454181134 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#272: UpgradeProcessorInternal@1434636325 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#273: UpgradeProcessorInternal@1318115128 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6c092b03:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:42084] msgSent=17154 msgReceived=980
processor#274: UpgradeProcessorInternal@63391341 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#275: UpgradeProcessorInternal@1941813270 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1202bac8:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50836] msgSent=8979 msgReceived=1442
processor#276: UpgradeProcessorInternal@1709007472 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#277: UpgradeProcessorInternal@1687754548 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#278: UpgradeProcessorInternal@1353840384 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#279: UpgradeProcessorInternal@689966738 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#280: UpgradeProcessorInternal@769715798 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@71f5b17e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48881] msgSent=5 msgReceived=8
processor#281: UpgradeProcessorInternal@538693548 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#282: UpgradeProcessorInternal@99010652 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#283: UpgradeProcessorInternal@46163835 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#284: UpgradeProcessorInternal@1653861160 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#285: UpgradeProcessorInternal@835521137 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#286: UpgradeProcessorInternal@1819046354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#287: UpgradeProcessorInternal@704460355 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1329f2a4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57088] msgSent=16877 msgReceived=962
processor#288: UpgradeProcessorInternal@999693398 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#289: UpgradeProcessorInternal@952037244 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#290: UpgradeProcessorInternal@1834838687 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#291: UpgradeProcessorInternal@1353907057 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#292: UpgradeProcessorInternal@1920282040 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#293: UpgradeProcessorInternal@502347326 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#294: UpgradeProcessorInternal@521254110 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#295: UpgradeProcessorInternal@642772382 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#296: UpgradeProcessorInternal@785858310 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
processor#297: UpgradeProcessorInternal@1208022481 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@781a5539:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53273] msgSent=1748 msgReceived=1047
processor#298: UpgradeProcessorInternal@2131665626 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@3b699188:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50660] msgSent=275 msgReceived=428
processor#299: UpgradeProcessorInternal@969235472 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#300: UpgradeProcessorInternal@632167035 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@138c331e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59302] msgSent=1638 msgReceived=830
processor#301: UpgradeProcessorInternal@344672093 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#302: UpgradeProcessorInternal@444927827 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#303: UpgradeProcessorInternal@826205929 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#304: UpgradeProcessorInternal@8738130 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#305: UpgradeProcessorInternal@1324275001 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@3dc58c00:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:52942] msgSent=48 msgReceived=13
processor#306: UpgradeProcessorInternal@1327508728 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
processor#307: UpgradeProcessorInternal@1235248762 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#308: UpgradeProcessorInternal@757308671 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#309: UpgradeProcessorInternal@394008226 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#310: UpgradeProcessorInternal@1948325825 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#311: UpgradeProcessorInternal@1802817168 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#312: UpgradeProcessorInternal@621216482 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#313: UpgradeProcessorInternal@1786662299 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@377ebd56:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:49470] msgSent=810 msgReceived=966
processor#314: UpgradeProcessorInternal@1842470969 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2bc4e23a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50677] msgSent=809 msgReceived=953
processor#315: UpgradeProcessorInternal@1885340041 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#316: UpgradeProcessorInternal@836220989 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#317: UpgradeProcessorInternal@446331766 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
processor#318: UpgradeProcessorInternal@352928484 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#319: UpgradeProcessorInternal@1949042653 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@31b182eb:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:55365] msgSent=611 msgReceived=662
processor#320: UpgradeProcessorInternal@1970768002 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
processor#321: UpgradeProcessorInternal@1914450919 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@506ac11a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53360] msgSent=19626 msgReceived=550
processor#322: UpgradeProcessorInternal@553558786 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#323: UpgradeProcessorInternal@648152412 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
processor#324: UpgradeProcessorInternal@157417119 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
processor#325: UpgradeProcessorInternal@794632354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
Nuof not connected: 291
Nuof connected: 34
webSocketMessageBrokerStats: WebSocketSession[33 current WS(33)-HttpStream(0)-HttpPoll(0), 2177 total, 0 closed abnormally (0 connect failure, 0 send limit, 251 transport error)]
stompSubProtocol[processed CONNECT(2116)-CONNECTED(2116)-DISCONNECT(0)]
stompBrokerRelay[null]
inboundChannel[pool size = 64, active threads = 0, queued tasks = 0, completed tasks = 542280]
outboundChannel[pool size = 64, active threads = 0, queued tasks = 0, completed tasks = 2183026]
sockJsScheduler[pool size = 32, active threads = 0, queued tasks = 9, completed tasks = 19964]
subProtocolWebSocketHandler: SubProtocolWebSocketHandler[StompSubProtocolHandler[v10.stomp, v11.stomp, v12.stomp]]
subProtocolWebSocketHandler.sessions: 33
Re: Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75
Posted by ma...@datenwort.at.INVALID.
I also spent quite some time already to create such a test case, but did not yet manage to find something.
For now it works in Production to remove those dangling processes once per hour (via bad reflection stuff ;-) ) … will try to add some logging to probably get a grasp about what is going on.
Best regards,
Mario
> Am 27.07.2023 um 12:45 schrieb Mark Thomas <ma...@apache.org>:
>
> I've taken a look at the code and can't see how this might be happening. I think a reproducible test case is going to be required to investigate this further.
>
> Mark
>
>
> On 12/07/2023 09:25, Mark Thomas wrote:
>> Hi Mario,
>> That does look like a possible bug.
>> I'll try and do a code review before the next release but from experience f you are able to figure out how to reproduce it that would help a lot.
>> Thanks,
>> Mark
>> On 06/07/2023 15:19, mario@datenwort.at.INVALID wrote:
>>> Hello!
>>>
>>> I guess I found a memory leak which I am not quite sure how to reproduce yet.
>>> Our application is using WebSockets and sometimes it seems that Tomcat 9.0.75 does not always remove the „UpgradeProcessorInternal“ process from the list of „waitingProcessors“ in „org.apache.coyote.AbstractProtocol“.
>>>
>>> I managed to create a dump (in that e-mail below) of that data from a live VM and found that some of those UpgradeProcessorInternal instances are referencing a "Closed Socket", which somehow feels odd to me. This list slowly grows. The entries are never removed.
>>> Those UpgradeProcessorInternal-s referencing a live socket are matching exactly the number of live WebSocket sessions.
>>>
>>> What do you think?
>>>
>>> Env:
>>> Tomcat: 9.0.75.0
>>> Spring Boot: 2.7.12
>>> Java: 17.0.7
>>>
>>>
>>> Best regards,
>>> Mario
>>>
>>>
>>> connector :Connector[HTTP/1.1-2888]
>>> waitingProcessors: 325
>>> processor#1: UpgradeProcessorInternal@1087825622 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#2: UpgradeProcessorInternal@92090237 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#3: UpgradeProcessorInternal@1641815002 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#4: UpgradeProcessorInternal@1749164616 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#5: UpgradeProcessorInternal@508142663 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#6: UpgradeProcessorInternal@1482593872 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#7: UpgradeProcessorInternal@1176145940 socketWrapper state: Closed NioChannel msgSent=4389 msgReceived=205
>>> processor#8: UpgradeProcessorInternal@1420482234 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#9: UpgradeProcessorInternal@31846900 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#10: UpgradeProcessorInternal@1726295282 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#11: UpgradeProcessorInternal@288536869 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#12: UpgradeProcessorInternal@969872343 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#13: UpgradeProcessorInternal@420227862 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#14: UpgradeProcessorInternal@552002299 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#15: UpgradeProcessorInternal@255507237 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#16: UpgradeProcessorInternal@724602640 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#17: UpgradeProcessorInternal@1706491794 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#18: UpgradeProcessorInternal@1925947118 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#19: UpgradeProcessorInternal@2044888516 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#20: UpgradeProcessorInternal@905596380 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#21: UpgradeProcessorInternal@897538387 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@61aa8978:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:40566] msgSent=7277 msgReceived=447
>>> processor#22: UpgradeProcessorInternal@1070005238 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#23: UpgradeProcessorInternal@34919463 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#24: UpgradeProcessorInternal@565924232 socketWrapper state: Closed NioChannel msgSent=12 msgReceived=14
>>> processor#25: UpgradeProcessorInternal@1146279009 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#26: UpgradeProcessorInternal@1467208007 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#27: UpgradeProcessorInternal@348911347 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#28: UpgradeProcessorInternal@38694518 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#29: UpgradeProcessorInternal@1164156250 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#30: UpgradeProcessorInternal@1675936734 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#31: UpgradeProcessorInternal@441511021 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#32: UpgradeProcessorInternal@59206586 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@5910f8ae:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50444] msgSent=3684 msgReceived=449
>>> processor#33: UpgradeProcessorInternal@1027164423 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#34: UpgradeProcessorInternal@2101191939 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#35: UpgradeProcessorInternal@1598120319 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#36: UpgradeProcessorInternal@1601686839 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#37: UpgradeProcessorInternal@2131114820 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#38: UpgradeProcessorInternal@1695970133 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#39: UpgradeProcessorInternal@1088759468 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@626eafa5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58972] msgSent=727 msgReceived=868
>>> processor#40: UpgradeProcessorInternal@332728223 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#41: UpgradeProcessorInternal@221860722 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#42: UpgradeProcessorInternal@773355092 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#43: UpgradeProcessorInternal@826604304 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#44: UpgradeProcessorInternal@267595687 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6c19b8c1:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50565] msgSent=2968 msgReceived=387
>>> processor#45: UpgradeProcessorInternal@417381557 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#46: UpgradeProcessorInternal@1626896557 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#47: UpgradeProcessorInternal@1326188889 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#48: UpgradeProcessorInternal@1620503747 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#49: UpgradeProcessorInternal@1588637927 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1ce5fc5a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:33528] msgSent=5765 msgReceived=442
>>> processor#50: UpgradeProcessorInternal@148940473 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#51: UpgradeProcessorInternal@915453133 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#52: UpgradeProcessorInternal@369115230 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#53: UpgradeProcessorInternal@405977197 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#54: UpgradeProcessorInternal@47138960 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#55: UpgradeProcessorInternal@178012412 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@13f91fab:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48880] msgSent=5 msgReceived=11
>>> processor#56: UpgradeProcessorInternal@360143126 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#57: UpgradeProcessorInternal@1196637488 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#58: UpgradeProcessorInternal@1741581739 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2815c011:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59371] msgSent=8604 msgReceived=853
>>> processor#59: UpgradeProcessorInternal@1053645483 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#60: UpgradeProcessorInternal@490296158 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#61: UpgradeProcessorInternal@280948950 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#62: UpgradeProcessorInternal@1763242354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#63: UpgradeProcessorInternal@170476101 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#64: UpgradeProcessorInternal@791654722 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#65: UpgradeProcessorInternal@808226370 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#66: UpgradeProcessorInternal@325893404 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#67: UpgradeProcessorInternal@1205002145 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#68: UpgradeProcessorInternal@105139248 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#69: UpgradeProcessorInternal@835317180 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#70: UpgradeProcessorInternal@1110689861 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2e9114a0:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57172] msgSent=31182 msgReceived=737
>>> processor#71: UpgradeProcessorInternal@694149416 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#72: UpgradeProcessorInternal@1917387827 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#73: UpgradeProcessorInternal@1669990384 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#74: UpgradeProcessorInternal@482149574 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#75: UpgradeProcessorInternal@1630942539 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#76: UpgradeProcessorInternal@661587730 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#77: UpgradeProcessorInternal@334804362 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#78: UpgradeProcessorInternal@1847955546 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@f2be4c4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60237] msgSent=537 msgReceived=557
>>> processor#79: UpgradeProcessorInternal@961085898 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#80: UpgradeProcessorInternal@1015170564 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#81: UpgradeProcessorInternal@1472660801 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#82: UpgradeProcessorInternal@1662742940 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#83: UpgradeProcessorInternal@783736881 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#84: UpgradeProcessorInternal@1033748247 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#85: UpgradeProcessorInternal@1631720904 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#86: UpgradeProcessorInternal@377835540 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
>>> processor#87: UpgradeProcessorInternal@208792804 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#88: UpgradeProcessorInternal@27480885 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#89: UpgradeProcessorInternal@6538996 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#90: UpgradeProcessorInternal@2030787475 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#91: UpgradeProcessorInternal@1780923582 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#92: UpgradeProcessorInternal@30596938 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#93: UpgradeProcessorInternal@165266752 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#94: UpgradeProcessorInternal@1937101292 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#95: UpgradeProcessorInternal@854833769 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#96: UpgradeProcessorInternal@1653312022 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#97: UpgradeProcessorInternal@946062077 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#98: UpgradeProcessorInternal@111135744 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#99: UpgradeProcessorInternal@1407359357 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@50247a32:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56550] msgSent=17041 msgReceived=1054
>>> processor#100: UpgradeProcessorInternal@1688084542 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#101: UpgradeProcessorInternal@1506439018 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#102: UpgradeProcessorInternal@515959904 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#103: UpgradeProcessorInternal@259281365 socketWrapper state: Closed NioChannel msgSent=12 msgReceived=12
>>> processor#104: UpgradeProcessorInternal@364095762 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#105: UpgradeProcessorInternal@307308786 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#106: UpgradeProcessorInternal@1677683546 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#107: UpgradeProcessorInternal@318642779 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#108: UpgradeProcessorInternal@914880558 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#109: UpgradeProcessorInternal@291835852 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#110: UpgradeProcessorInternal@168338082 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#111: UpgradeProcessorInternal@1174282578 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#112: UpgradeProcessorInternal@331106577 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#113: UpgradeProcessorInternal@1617882333 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#114: UpgradeProcessorInternal@542714593 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#115: UpgradeProcessorInternal@1156269649 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#116: UpgradeProcessorInternal@201406651 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#117: UpgradeProcessorInternal@2058862092 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#118: UpgradeProcessorInternal@857502172 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#119: UpgradeProcessorInternal@996422055 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#120: UpgradeProcessorInternal@1815696637 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#121: UpgradeProcessorInternal@736438049 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@7b35f1fe:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58899] msgSent=799 msgReceived=935
>>> processor#122: UpgradeProcessorInternal@477398194 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#123: UpgradeProcessorInternal@1219089519 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#124: UpgradeProcessorInternal@1272242963 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#125: UpgradeProcessorInternal@121254386 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#126: UpgradeProcessorInternal@1669364040 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#127: UpgradeProcessorInternal@275174062 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#128: UpgradeProcessorInternal@554430912 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#129: UpgradeProcessorInternal@1187725828 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#130: UpgradeProcessorInternal@1531691908 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#131: UpgradeProcessorInternal@446727798 socketWrapper state: Closed NioChannel msgSent=283 msgReceived=259
>>> processor#132: UpgradeProcessorInternal@362658123 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#133: UpgradeProcessorInternal@330299253 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#134: UpgradeProcessorInternal@1417579684 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#135: UpgradeProcessorInternal@79366241 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#136: UpgradeProcessorInternal@1794180138 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#137: UpgradeProcessorInternal@773525189 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#138: UpgradeProcessorInternal@88744340 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#139: UpgradeProcessorInternal@709557909 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#140: UpgradeProcessorInternal@263285102 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#141: UpgradeProcessorInternal@1223181320 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#142: UpgradeProcessorInternal@1288060966 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@106e2540:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:61373] msgSent=9780 msgReceived=297
>>> processor#143: UpgradeProcessorInternal@102971590 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#144: UpgradeProcessorInternal@808226506 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#145: UpgradeProcessorInternal@1228942248 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#146: UpgradeProcessorInternal@1214228151 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#147: UpgradeProcessorInternal@550718522 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#148: UpgradeProcessorInternal@546382458 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#149: UpgradeProcessorInternal@430113102 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#150: UpgradeProcessorInternal@1801235373 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#151: UpgradeProcessorInternal@613091448 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#152: UpgradeProcessorInternal@567610662 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@7d797520:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:32782] msgSent=7435 msgReceived=471
>>> processor#153: UpgradeProcessorInternal@864215926 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#154: UpgradeProcessorInternal@257132453 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#155: UpgradeProcessorInternal@1679461613 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#156: UpgradeProcessorInternal@1792036392 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#157: UpgradeProcessorInternal@1237165381 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#158: UpgradeProcessorInternal@2095009319 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#159: UpgradeProcessorInternal@405187807 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#160: UpgradeProcessorInternal@1137415986 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#161: UpgradeProcessorInternal@1259367401 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#162: UpgradeProcessorInternal@590114774 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#163: UpgradeProcessorInternal@1979332358 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#164: UpgradeProcessorInternal@1910582044 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#165: UpgradeProcessorInternal@1205302056 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#166: UpgradeProcessorInternal@1454816183 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#167: UpgradeProcessorInternal@1127766585 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#168: UpgradeProcessorInternal@1563629106 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2cdc91f6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59195] msgSent=832 msgReceived=865
>>> processor#169: UpgradeProcessorInternal@27650727 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#170: UpgradeProcessorInternal@1847115547 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#171: UpgradeProcessorInternal@1939276436 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#172: UpgradeProcessorInternal@2038944386 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#173: UpgradeProcessorInternal@1155354584 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#174: UpgradeProcessorInternal@97303750 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#175: UpgradeProcessorInternal@1608676584 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#176: UpgradeProcessorInternal@886887382 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#177: UpgradeProcessorInternal@620014078 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#178: UpgradeProcessorInternal@551004124 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#179: UpgradeProcessorInternal@2110011592 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#180: UpgradeProcessorInternal@1908346035 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#181: UpgradeProcessorInternal@378148743 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#182: UpgradeProcessorInternal@1156173823 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#183: UpgradeProcessorInternal@64974024 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#184: UpgradeProcessorInternal@1009466172 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#185: UpgradeProcessorInternal@536984857 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#186: UpgradeProcessorInternal@383186893 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#187: UpgradeProcessorInternal@836747459 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#188: UpgradeProcessorInternal@1077816609 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#189: UpgradeProcessorInternal@645247338 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=10
>>> processor#190: UpgradeProcessorInternal@921863122 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#191: UpgradeProcessorInternal@1763449404 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#192: UpgradeProcessorInternal@464409230 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@519deb3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:36638] msgSent=5311 msgReceived=430
>>> processor#193: UpgradeProcessorInternal@1278303507 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#194: UpgradeProcessorInternal@1244687124 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#195: UpgradeProcessorInternal@1930251816 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#196: UpgradeProcessorInternal@43494834 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@29f70e25:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:51288] msgSent=7 msgReceived=9
>>> processor#197: UpgradeProcessorInternal@1612242224 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#198: UpgradeProcessorInternal@1831479299 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#199: UpgradeProcessorInternal@287185460 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#200: UpgradeProcessorInternal@1947436860 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#201: UpgradeProcessorInternal@171168002 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#202: UpgradeProcessorInternal@1436058281 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#203: UpgradeProcessorInternal@1647228700 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#204: UpgradeProcessorInternal@658189327 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#205: UpgradeProcessorInternal@1207405250 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#206: UpgradeProcessorInternal@1206888150 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#207: UpgradeProcessorInternal@11407255 socketWrapper state: Closed NioChannel msgSent=512 msgReceived=549
>>> processor#208: UpgradeProcessorInternal@999984803 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6887705e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59146] msgSent=719 msgReceived=796
>>> processor#209: UpgradeProcessorInternal@2079580872 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#210: UpgradeProcessorInternal@26762917 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#211: UpgradeProcessorInternal@653767625 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#212: UpgradeProcessorInternal@685357542 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#213: UpgradeProcessorInternal@2105023035 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#214: UpgradeProcessorInternal@906125638 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#215: UpgradeProcessorInternal@1739573994 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#216: UpgradeProcessorInternal@500242071 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#217: UpgradeProcessorInternal@1067985634 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#218: UpgradeProcessorInternal@1526977615 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@113386a5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56957] msgSent=11502 msgReceived=275
>>> processor#219: UpgradeProcessorInternal@1564249713 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#220: UpgradeProcessorInternal@1271579268 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#221: UpgradeProcessorInternal@1251932623 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#222: UpgradeProcessorInternal@336201051 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#223: UpgradeProcessorInternal@1969809982 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#224: UpgradeProcessorInternal@1001859809 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#225: UpgradeProcessorInternal@1073172128 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#226: UpgradeProcessorInternal@787383222 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#227: UpgradeProcessorInternal@599435490 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#228: UpgradeProcessorInternal@202250581 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#229: UpgradeProcessorInternal@2120255293 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#230: UpgradeProcessorInternal@1888709582 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#231: UpgradeProcessorInternal@765967098 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#232: UpgradeProcessorInternal@1517419823 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#233: UpgradeProcessorInternal@1701040641 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#234: UpgradeProcessorInternal@1146615605 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#235: UpgradeProcessorInternal@117512293 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#236: UpgradeProcessorInternal@2012342422 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#237: UpgradeProcessorInternal@1973886667 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#238: UpgradeProcessorInternal@1105514123 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#239: UpgradeProcessorInternal@1632774205 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2e335b3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60332] msgSent=1124 msgReceived=510
>>> processor#240: UpgradeProcessorInternal@1643707528 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#241: UpgradeProcessorInternal@1788721645 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#242: UpgradeProcessorInternal@1338930879 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#243: UpgradeProcessorInternal@1437970118 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#244: UpgradeProcessorInternal@825036894 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#245: UpgradeProcessorInternal@1411771222 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#246: UpgradeProcessorInternal@1641563307 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#247: UpgradeProcessorInternal@1910767760 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#248: UpgradeProcessorInternal@1376679291 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#249: UpgradeProcessorInternal@1369067244 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#250: UpgradeProcessorInternal@640417107 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@12ac4472:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56706] msgSent=725 msgReceived=838
>>> processor#251: UpgradeProcessorInternal@1723934650 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#252: UpgradeProcessorInternal@390093373 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#253: UpgradeProcessorInternal@1679102358 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#254: UpgradeProcessorInternal@1943695966 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#255: UpgradeProcessorInternal@1675844197 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@46028bc6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59060] msgSent=1799 msgReceived=1047
>>> processor#256: UpgradeProcessorInternal@1597795515 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#257: UpgradeProcessorInternal@449048399 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#258: UpgradeProcessorInternal@1138357335 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#259: UpgradeProcessorInternal@1150701318 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#260: UpgradeProcessorInternal@759289043 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#261: UpgradeProcessorInternal@663090198 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#262: UpgradeProcessorInternal@1804724738 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#263: UpgradeProcessorInternal@1681171367 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#264: UpgradeProcessorInternal@894140638 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#265: UpgradeProcessorInternal@1268650502 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#266: UpgradeProcessorInternal@201391001 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
>>> processor#267: UpgradeProcessorInternal@4454872 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@deddb3c:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50534] msgSent=2955 msgReceived=422
>>> processor#268: UpgradeProcessorInternal@1292060318 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#269: UpgradeProcessorInternal@1799841497 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#270: UpgradeProcessorInternal@1293709 socketWrapper state: Closed NioChannel msgSent=9 msgReceived=9
>>> processor#271: UpgradeProcessorInternal@1454181134 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#272: UpgradeProcessorInternal@1434636325 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#273: UpgradeProcessorInternal@1318115128 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@6c092b03:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:42084] msgSent=17154 msgReceived=980
>>> processor#274: UpgradeProcessorInternal@63391341 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#275: UpgradeProcessorInternal@1941813270 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1202bac8:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50836] msgSent=8979 msgReceived=1442
>>> processor#276: UpgradeProcessorInternal@1709007472 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#277: UpgradeProcessorInternal@1687754548 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#278: UpgradeProcessorInternal@1353840384 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#279: UpgradeProcessorInternal@689966738 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#280: UpgradeProcessorInternal@769715798 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@71f5b17e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48881] msgSent=5 msgReceived=8
>>> processor#281: UpgradeProcessorInternal@538693548 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#282: UpgradeProcessorInternal@99010652 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#283: UpgradeProcessorInternal@46163835 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#284: UpgradeProcessorInternal@1653861160 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#285: UpgradeProcessorInternal@835521137 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#286: UpgradeProcessorInternal@1819046354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#287: UpgradeProcessorInternal@704460355 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@1329f2a4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57088] msgSent=16877 msgReceived=962
>>> processor#288: UpgradeProcessorInternal@999693398 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#289: UpgradeProcessorInternal@952037244 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#290: UpgradeProcessorInternal@1834838687 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#291: UpgradeProcessorInternal@1353907057 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#292: UpgradeProcessorInternal@1920282040 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#293: UpgradeProcessorInternal@502347326 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#294: UpgradeProcessorInternal@521254110 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#295: UpgradeProcessorInternal@642772382 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#296: UpgradeProcessorInternal@785858310 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=13
>>> processor#297: UpgradeProcessorInternal@1208022481 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@781a5539:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53273] msgSent=1748 msgReceived=1047
>>> processor#298: UpgradeProcessorInternal@2131665626 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@3b699188:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50660] msgSent=275 msgReceived=428
>>> processor#299: UpgradeProcessorInternal@969235472 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#300: UpgradeProcessorInternal@632167035 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@138c331e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59302] msgSent=1638 msgReceived=830
>>> processor#301: UpgradeProcessorInternal@344672093 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#302: UpgradeProcessorInternal@444927827 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#303: UpgradeProcessorInternal@826205929 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#304: UpgradeProcessorInternal@8738130 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#305: UpgradeProcessorInternal@1324275001 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@3dc58c00:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:52942] msgSent=48 msgReceived=13
>>> processor#306: UpgradeProcessorInternal@1327508728 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> processor#307: UpgradeProcessorInternal@1235248762 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#308: UpgradeProcessorInternal@757308671 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#309: UpgradeProcessorInternal@394008226 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#310: UpgradeProcessorInternal@1948325825 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#311: UpgradeProcessorInternal@1802817168 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#312: UpgradeProcessorInternal@621216482 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#313: UpgradeProcessorInternal@1786662299 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@377ebd56:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:49470] msgSent=810 msgReceived=966
>>> processor#314: UpgradeProcessorInternal@1842470969 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@2bc4e23a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50677] msgSent=809 msgReceived=953
>>> processor#315: UpgradeProcessorInternal@1885340041 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#316: UpgradeProcessorInternal@836220989 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#317: UpgradeProcessorInternal@446331766 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=9
>>> processor#318: UpgradeProcessorInternal@352928484 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#319: UpgradeProcessorInternal@1949042653 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@31b182eb:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:55365] msgSent=611 msgReceived=662
>>> processor#320: UpgradeProcessorInternal@1970768002 socketWrapper state: Closed NioChannel msgSent=11 msgReceived=12
>>> processor#321: UpgradeProcessorInternal@1914450919 socketWrapper state: org.apache.tomcat.util.net.SecureNioChannel@506ac11a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53360] msgSent=19626 msgReceived=550
>>> processor#322: UpgradeProcessorInternal@553558786 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#323: UpgradeProcessorInternal@648152412 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=9
>>> processor#324: UpgradeProcessorInternal@157417119 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=13
>>> processor#325: UpgradeProcessorInternal@794632354 socketWrapper state: Closed NioChannel msgSent=10 msgReceived=12
>>> Nuof not connected: 291
>>> Nuof connected: 34
>>>
>>>
>>> webSocketMessageBrokerStats: WebSocketSession[33 current WS(33)-HttpStream(0)-HttpPoll(0), 2177 total, 0 closed abnormally (0 connect failure, 0 send limit, 251 transport error)]
>>> stompSubProtocol[processed CONNECT(2116)-CONNECTED(2116)-DISCONNECT(0)]
>>> stompBrokerRelay[null]
>>> inboundChannel[pool size = 64, active threads = 0, queued tasks = 0, completed tasks = 542280]
>>> outboundChannel[pool size = 64, active threads = 0, queued tasks = 0, completed tasks = 2183026]
>>> sockJsScheduler[pool size = 32, active threads = 0, queued tasks = 9, completed tasks = 19964]
>>> subProtocolWebSocketHandler: SubProtocolWebSocketHandler[StompSubProtocolHandler[v10.stomp, v11.stomp, v12.stomp]]
>>> subProtocolWebSocketHandler.sessions: 33
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75
Posted by Mark Thomas <ma...@apache.org>.
I've taken a look at the code and can't see how this might be happening.
I think a reproducible test case is going to be required to investigate
this further.
Mark
On 12/07/2023 09:25, Mark Thomas wrote:
> Hi Mario,
>
> That does look like a possible bug.
>
> I'll try and do a code review before the next release but from
> experience f you are able to figure out how to reproduce it that would
> help a lot.
>
> Thanks,
>
> Mark
>
>
> On 06/07/2023 15:19, mario@datenwort.at.INVALID wrote:
>> Hello!
>>
>> I guess I found a memory leak which I am not quite sure how to
>> reproduce yet.
>> Our application is using WebSockets and sometimes it seems that Tomcat
>> 9.0.75 does not always remove the „UpgradeProcessorInternal“ process
>> from the list of „waitingProcessors“ in
>> „org.apache.coyote.AbstractProtocol“.
>>
>> I managed to create a dump (in that e-mail below) of that data from a
>> live VM and found that some of those UpgradeProcessorInternal
>> instances are referencing a "Closed Socket", which somehow feels odd
>> to me. This list slowly grows. The entries are never removed.
>> Those UpgradeProcessorInternal-s referencing a live socket are
>> matching exactly the number of live WebSocket sessions.
>>
>> What do you think?
>>
>> Env:
>> Tomcat: 9.0.75.0
>> Spring Boot: 2.7.12
>> Java: 17.0.7
>>
>>
>> Best regards,
>> Mario
>>
>>
>> connector :Connector[HTTP/1.1-2888]
>> waitingProcessors: 325
>> processor#1: UpgradeProcessorInternal@1087825622 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=13
>> processor#2: UpgradeProcessorInternal@92090237 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#3: UpgradeProcessorInternal@1641815002 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#4: UpgradeProcessorInternal@1749164616 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#5: UpgradeProcessorInternal@508142663 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#6: UpgradeProcessorInternal@1482593872 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#7: UpgradeProcessorInternal@1176145940 socketWrapper state:
>> Closed NioChannel msgSent=4389 msgReceived=205
>> processor#8: UpgradeProcessorInternal@1420482234 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#9: UpgradeProcessorInternal@31846900 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#10: UpgradeProcessorInternal@1726295282 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#11: UpgradeProcessorInternal@288536869 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#12: UpgradeProcessorInternal@969872343 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#13: UpgradeProcessorInternal@420227862 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#14: UpgradeProcessorInternal@552002299 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#15: UpgradeProcessorInternal@255507237 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#16: UpgradeProcessorInternal@724602640 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#17: UpgradeProcessorInternal@1706491794 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#18: UpgradeProcessorInternal@1925947118 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#19: UpgradeProcessorInternal@2044888516 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#20: UpgradeProcessorInternal@905596380 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#21: UpgradeProcessorInternal@897538387 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@61aa8978:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:40566] msgSent=7277 msgReceived=447
>> processor#22: UpgradeProcessorInternal@1070005238 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#23: UpgradeProcessorInternal@34919463 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#24: UpgradeProcessorInternal@565924232 socketWrapper state:
>> Closed NioChannel msgSent=12 msgReceived=14
>> processor#25: UpgradeProcessorInternal@1146279009 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#26: UpgradeProcessorInternal@1467208007 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#27: UpgradeProcessorInternal@348911347 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#28: UpgradeProcessorInternal@38694518 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#29: UpgradeProcessorInternal@1164156250 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#30: UpgradeProcessorInternal@1675936734 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#31: UpgradeProcessorInternal@441511021 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#32: UpgradeProcessorInternal@59206586 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@5910f8ae:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50444] msgSent=3684 msgReceived=449
>> processor#33: UpgradeProcessorInternal@1027164423 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#34: UpgradeProcessorInternal@2101191939 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#35: UpgradeProcessorInternal@1598120319 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#36: UpgradeProcessorInternal@1601686839 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#37: UpgradeProcessorInternal@2131114820 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#38: UpgradeProcessorInternal@1695970133 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#39: UpgradeProcessorInternal@1088759468 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@626eafa5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58972] msgSent=727 msgReceived=868
>> processor#40: UpgradeProcessorInternal@332728223 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#41: UpgradeProcessorInternal@221860722 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#42: UpgradeProcessorInternal@773355092 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#43: UpgradeProcessorInternal@826604304 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#44: UpgradeProcessorInternal@267595687 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@6c19b8c1:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50565] msgSent=2968 msgReceived=387
>> processor#45: UpgradeProcessorInternal@417381557 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#46: UpgradeProcessorInternal@1626896557 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#47: UpgradeProcessorInternal@1326188889 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#48: UpgradeProcessorInternal@1620503747 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#49: UpgradeProcessorInternal@1588637927 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@1ce5fc5a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:33528] msgSent=5765 msgReceived=442
>> processor#50: UpgradeProcessorInternal@148940473 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#51: UpgradeProcessorInternal@915453133 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#52: UpgradeProcessorInternal@369115230 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#53: UpgradeProcessorInternal@405977197 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#54: UpgradeProcessorInternal@47138960 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#55: UpgradeProcessorInternal@178012412 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@13f91fab:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48880] msgSent=5 msgReceived=11
>> processor#56: UpgradeProcessorInternal@360143126 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#57: UpgradeProcessorInternal@1196637488 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#58: UpgradeProcessorInternal@1741581739 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@2815c011:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59371] msgSent=8604 msgReceived=853
>> processor#59: UpgradeProcessorInternal@1053645483 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#60: UpgradeProcessorInternal@490296158 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#61: UpgradeProcessorInternal@280948950 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#62: UpgradeProcessorInternal@1763242354 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#63: UpgradeProcessorInternal@170476101 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#64: UpgradeProcessorInternal@791654722 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#65: UpgradeProcessorInternal@808226370 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#66: UpgradeProcessorInternal@325893404 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#67: UpgradeProcessorInternal@1205002145 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#68: UpgradeProcessorInternal@105139248 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#69: UpgradeProcessorInternal@835317180 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#70: UpgradeProcessorInternal@1110689861 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@2e9114a0:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57172] msgSent=31182 msgReceived=737
>> processor#71: UpgradeProcessorInternal@694149416 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=13
>> processor#72: UpgradeProcessorInternal@1917387827 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#73: UpgradeProcessorInternal@1669990384 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#74: UpgradeProcessorInternal@482149574 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#75: UpgradeProcessorInternal@1630942539 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#76: UpgradeProcessorInternal@661587730 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#77: UpgradeProcessorInternal@334804362 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#78: UpgradeProcessorInternal@1847955546 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@f2be4c4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60237] msgSent=537 msgReceived=557
>> processor#79: UpgradeProcessorInternal@961085898 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#80: UpgradeProcessorInternal@1015170564 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#81: UpgradeProcessorInternal@1472660801 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#82: UpgradeProcessorInternal@1662742940 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#83: UpgradeProcessorInternal@783736881 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#84: UpgradeProcessorInternal@1033748247 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#85: UpgradeProcessorInternal@1631720904 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#86: UpgradeProcessorInternal@377835540 socketWrapper state:
>> Closed NioChannel msgSent=9 msgReceived=9
>> processor#87: UpgradeProcessorInternal@208792804 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#88: UpgradeProcessorInternal@27480885 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#89: UpgradeProcessorInternal@6538996 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#90: UpgradeProcessorInternal@2030787475 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#91: UpgradeProcessorInternal@1780923582 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#92: UpgradeProcessorInternal@30596938 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#93: UpgradeProcessorInternal@165266752 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#94: UpgradeProcessorInternal@1937101292 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#95: UpgradeProcessorInternal@854833769 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#96: UpgradeProcessorInternal@1653312022 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#97: UpgradeProcessorInternal@946062077 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#98: UpgradeProcessorInternal@111135744 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#99: UpgradeProcessorInternal@1407359357 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@50247a32:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56550] msgSent=17041 msgReceived=1054
>> processor#100: UpgradeProcessorInternal@1688084542 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#101: UpgradeProcessorInternal@1506439018 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#102: UpgradeProcessorInternal@515959904 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#103: UpgradeProcessorInternal@259281365 socketWrapper state:
>> Closed NioChannel msgSent=12 msgReceived=12
>> processor#104: UpgradeProcessorInternal@364095762 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#105: UpgradeProcessorInternal@307308786 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#106: UpgradeProcessorInternal@1677683546 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#107: UpgradeProcessorInternal@318642779 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#108: UpgradeProcessorInternal@914880558 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#109: UpgradeProcessorInternal@291835852 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#110: UpgradeProcessorInternal@168338082 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#111: UpgradeProcessorInternal@1174282578 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#112: UpgradeProcessorInternal@331106577 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#113: UpgradeProcessorInternal@1617882333 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#114: UpgradeProcessorInternal@542714593 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#115: UpgradeProcessorInternal@1156269649 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=13
>> processor#116: UpgradeProcessorInternal@201406651 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#117: UpgradeProcessorInternal@2058862092 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#118: UpgradeProcessorInternal@857502172 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#119: UpgradeProcessorInternal@996422055 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#120: UpgradeProcessorInternal@1815696637 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#121: UpgradeProcessorInternal@736438049 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@7b35f1fe:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58899] msgSent=799 msgReceived=935
>> processor#122: UpgradeProcessorInternal@477398194 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#123: UpgradeProcessorInternal@1219089519 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#124: UpgradeProcessorInternal@1272242963 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#125: UpgradeProcessorInternal@121254386 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#126: UpgradeProcessorInternal@1669364040 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#127: UpgradeProcessorInternal@275174062 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#128: UpgradeProcessorInternal@554430912 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#129: UpgradeProcessorInternal@1187725828 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#130: UpgradeProcessorInternal@1531691908 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#131: UpgradeProcessorInternal@446727798 socketWrapper state:
>> Closed NioChannel msgSent=283 msgReceived=259
>> processor#132: UpgradeProcessorInternal@362658123 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#133: UpgradeProcessorInternal@330299253 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#134: UpgradeProcessorInternal@1417579684 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#135: UpgradeProcessorInternal@79366241 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#136: UpgradeProcessorInternal@1794180138 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#137: UpgradeProcessorInternal@773525189 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#138: UpgradeProcessorInternal@88744340 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#139: UpgradeProcessorInternal@709557909 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#140: UpgradeProcessorInternal@263285102 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#141: UpgradeProcessorInternal@1223181320 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#142: UpgradeProcessorInternal@1288060966 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@106e2540:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:61373] msgSent=9780 msgReceived=297
>> processor#143: UpgradeProcessorInternal@102971590 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#144: UpgradeProcessorInternal@808226506 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#145: UpgradeProcessorInternal@1228942248 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#146: UpgradeProcessorInternal@1214228151 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#147: UpgradeProcessorInternal@550718522 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#148: UpgradeProcessorInternal@546382458 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#149: UpgradeProcessorInternal@430113102 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#150: UpgradeProcessorInternal@1801235373 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#151: UpgradeProcessorInternal@613091448 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#152: UpgradeProcessorInternal@567610662 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@7d797520:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:32782] msgSent=7435 msgReceived=471
>> processor#153: UpgradeProcessorInternal@864215926 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#154: UpgradeProcessorInternal@257132453 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#155: UpgradeProcessorInternal@1679461613 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#156: UpgradeProcessorInternal@1792036392 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#157: UpgradeProcessorInternal@1237165381 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#158: UpgradeProcessorInternal@2095009319 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#159: UpgradeProcessorInternal@405187807 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#160: UpgradeProcessorInternal@1137415986 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#161: UpgradeProcessorInternal@1259367401 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#162: UpgradeProcessorInternal@590114774 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#163: UpgradeProcessorInternal@1979332358 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#164: UpgradeProcessorInternal@1910582044 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#165: UpgradeProcessorInternal@1205302056 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#166: UpgradeProcessorInternal@1454816183 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#167: UpgradeProcessorInternal@1127766585 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#168: UpgradeProcessorInternal@1563629106 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@2cdc91f6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59195] msgSent=832 msgReceived=865
>> processor#169: UpgradeProcessorInternal@27650727 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#170: UpgradeProcessorInternal@1847115547 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#171: UpgradeProcessorInternal@1939276436 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#172: UpgradeProcessorInternal@2038944386 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#173: UpgradeProcessorInternal@1155354584 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#174: UpgradeProcessorInternal@97303750 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#175: UpgradeProcessorInternal@1608676584 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#176: UpgradeProcessorInternal@886887382 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#177: UpgradeProcessorInternal@620014078 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#178: UpgradeProcessorInternal@551004124 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#179: UpgradeProcessorInternal@2110011592 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=13
>> processor#180: UpgradeProcessorInternal@1908346035 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#181: UpgradeProcessorInternal@378148743 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#182: UpgradeProcessorInternal@1156173823 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#183: UpgradeProcessorInternal@64974024 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#184: UpgradeProcessorInternal@1009466172 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#185: UpgradeProcessorInternal@536984857 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#186: UpgradeProcessorInternal@383186893 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#187: UpgradeProcessorInternal@836747459 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#188: UpgradeProcessorInternal@1077816609 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#189: UpgradeProcessorInternal@645247338 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=10
>> processor#190: UpgradeProcessorInternal@921863122 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#191: UpgradeProcessorInternal@1763449404 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#192: UpgradeProcessorInternal@464409230 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@519deb3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:36638] msgSent=5311 msgReceived=430
>> processor#193: UpgradeProcessorInternal@1278303507 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#194: UpgradeProcessorInternal@1244687124 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#195: UpgradeProcessorInternal@1930251816 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#196: UpgradeProcessorInternal@43494834 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@29f70e25:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:51288] msgSent=7 msgReceived=9
>> processor#197: UpgradeProcessorInternal@1612242224 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#198: UpgradeProcessorInternal@1831479299 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#199: UpgradeProcessorInternal@287185460 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#200: UpgradeProcessorInternal@1947436860 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#201: UpgradeProcessorInternal@171168002 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#202: UpgradeProcessorInternal@1436058281 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#203: UpgradeProcessorInternal@1647228700 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#204: UpgradeProcessorInternal@658189327 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#205: UpgradeProcessorInternal@1207405250 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#206: UpgradeProcessorInternal@1206888150 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#207: UpgradeProcessorInternal@11407255 socketWrapper state:
>> Closed NioChannel msgSent=512 msgReceived=549
>> processor#208: UpgradeProcessorInternal@999984803 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@6887705e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59146] msgSent=719 msgReceived=796
>> processor#209: UpgradeProcessorInternal@2079580872 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#210: UpgradeProcessorInternal@26762917 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#211: UpgradeProcessorInternal@653767625 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#212: UpgradeProcessorInternal@685357542 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#213: UpgradeProcessorInternal@2105023035 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#214: UpgradeProcessorInternal@906125638 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#215: UpgradeProcessorInternal@1739573994 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#216: UpgradeProcessorInternal@500242071 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#217: UpgradeProcessorInternal@1067985634 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#218: UpgradeProcessorInternal@1526977615 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@113386a5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56957] msgSent=11502 msgReceived=275
>> processor#219: UpgradeProcessorInternal@1564249713 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#220: UpgradeProcessorInternal@1271579268 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#221: UpgradeProcessorInternal@1251932623 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#222: UpgradeProcessorInternal@336201051 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#223: UpgradeProcessorInternal@1969809982 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#224: UpgradeProcessorInternal@1001859809 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#225: UpgradeProcessorInternal@1073172128 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#226: UpgradeProcessorInternal@787383222 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#227: UpgradeProcessorInternal@599435490 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#228: UpgradeProcessorInternal@202250581 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#229: UpgradeProcessorInternal@2120255293 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#230: UpgradeProcessorInternal@1888709582 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#231: UpgradeProcessorInternal@765967098 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#232: UpgradeProcessorInternal@1517419823 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#233: UpgradeProcessorInternal@1701040641 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#234: UpgradeProcessorInternal@1146615605 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#235: UpgradeProcessorInternal@117512293 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#236: UpgradeProcessorInternal@2012342422 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#237: UpgradeProcessorInternal@1973886667 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#238: UpgradeProcessorInternal@1105514123 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#239: UpgradeProcessorInternal@1632774205 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@2e335b3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60332] msgSent=1124 msgReceived=510
>> processor#240: UpgradeProcessorInternal@1643707528 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#241: UpgradeProcessorInternal@1788721645 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#242: UpgradeProcessorInternal@1338930879 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#243: UpgradeProcessorInternal@1437970118 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#244: UpgradeProcessorInternal@825036894 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#245: UpgradeProcessorInternal@1411771222 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#246: UpgradeProcessorInternal@1641563307 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#247: UpgradeProcessorInternal@1910767760 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#248: UpgradeProcessorInternal@1376679291 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#249: UpgradeProcessorInternal@1369067244 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#250: UpgradeProcessorInternal@640417107 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@12ac4472:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56706] msgSent=725 msgReceived=838
>> processor#251: UpgradeProcessorInternal@1723934650 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#252: UpgradeProcessorInternal@390093373 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> processor#253: UpgradeProcessorInternal@1679102358 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#254: UpgradeProcessorInternal@1943695966 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#255: UpgradeProcessorInternal@1675844197 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@46028bc6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59060] msgSent=1799 msgReceived=1047
>> processor#256: UpgradeProcessorInternal@1597795515 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#257: UpgradeProcessorInternal@449048399 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#258: UpgradeProcessorInternal@1138357335 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#259: UpgradeProcessorInternal@1150701318 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#260: UpgradeProcessorInternal@759289043 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=13
>> processor#261: UpgradeProcessorInternal@663090198 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#262: UpgradeProcessorInternal@1804724738 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#263: UpgradeProcessorInternal@1681171367 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#264: UpgradeProcessorInternal@894140638 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#265: UpgradeProcessorInternal@1268650502 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=9
>> processor#266: UpgradeProcessorInternal@201391001 socketWrapper state:
>> Closed NioChannel msgSent=9 msgReceived=9
>> processor#267: UpgradeProcessorInternal@4454872 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@deddb3c:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50534] msgSent=2955 msgReceived=422
>> processor#268: UpgradeProcessorInternal@1292060318 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#269: UpgradeProcessorInternal@1799841497 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#270: UpgradeProcessorInternal@1293709 socketWrapper state:
>> Closed NioChannel msgSent=9 msgReceived=9
>> processor#271: UpgradeProcessorInternal@1454181134 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#272: UpgradeProcessorInternal@1434636325 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#273: UpgradeProcessorInternal@1318115128 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@6c092b03:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:42084] msgSent=17154 msgReceived=980
>> processor#274: UpgradeProcessorInternal@63391341 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#275: UpgradeProcessorInternal@1941813270 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@1202bac8:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50836] msgSent=8979 msgReceived=1442
>> processor#276: UpgradeProcessorInternal@1709007472 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#277: UpgradeProcessorInternal@1687754548 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#278: UpgradeProcessorInternal@1353840384 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#279: UpgradeProcessorInternal@689966738 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#280: UpgradeProcessorInternal@769715798 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@71f5b17e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48881] msgSent=5 msgReceived=8
>> processor#281: UpgradeProcessorInternal@538693548 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#282: UpgradeProcessorInternal@99010652 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#283: UpgradeProcessorInternal@46163835 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#284: UpgradeProcessorInternal@1653861160 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#285: UpgradeProcessorInternal@835521137 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=12
>> processor#286: UpgradeProcessorInternal@1819046354 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#287: UpgradeProcessorInternal@704460355 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@1329f2a4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57088] msgSent=16877 msgReceived=962
>> processor#288: UpgradeProcessorInternal@999693398 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=13
>> processor#289: UpgradeProcessorInternal@952037244 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#290: UpgradeProcessorInternal@1834838687 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#291: UpgradeProcessorInternal@1353907057 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#292: UpgradeProcessorInternal@1920282040 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#293: UpgradeProcessorInternal@502347326 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#294: UpgradeProcessorInternal@521254110 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#295: UpgradeProcessorInternal@642772382 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#296: UpgradeProcessorInternal@785858310 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=13
>> processor#297: UpgradeProcessorInternal@1208022481 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@781a5539:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53273] msgSent=1748 msgReceived=1047
>> processor#298: UpgradeProcessorInternal@2131665626 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@3b699188:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50660] msgSent=275 msgReceived=428
>> processor#299: UpgradeProcessorInternal@969235472 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#300: UpgradeProcessorInternal@632167035 socketWrapper state:
>> org.apache.tomcat.util.net.SecureNioChannel@138c331e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59302] msgSent=1638 msgReceived=830
>> processor#301: UpgradeProcessorInternal@344672093 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#302: UpgradeProcessorInternal@444927827 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#303: UpgradeProcessorInternal@826205929 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#304: UpgradeProcessorInternal@8738130 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#305: UpgradeProcessorInternal@1324275001 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@3dc58c00:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:52942] msgSent=48 msgReceived=13
>> processor#306: UpgradeProcessorInternal@1327508728 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=12
>> processor#307: UpgradeProcessorInternal@1235248762 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#308: UpgradeProcessorInternal@757308671 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#309: UpgradeProcessorInternal@394008226 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#310: UpgradeProcessorInternal@1948325825 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=9
>> processor#311: UpgradeProcessorInternal@1802817168 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#312: UpgradeProcessorInternal@621216482 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#313: UpgradeProcessorInternal@1786662299 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@377ebd56:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:49470] msgSent=810 msgReceived=966
>> processor#314: UpgradeProcessorInternal@1842470969 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@2bc4e23a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50677] msgSent=809 msgReceived=953
>> processor#315: UpgradeProcessorInternal@1885340041 socketWrapper
>> state: Closed NioChannel msgSent=10 msgReceived=13
>> processor#316: UpgradeProcessorInternal@836220989 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#317: UpgradeProcessorInternal@446331766 socketWrapper state:
>> Closed NioChannel msgSent=11 msgReceived=9
>> processor#318: UpgradeProcessorInternal@352928484 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#319: UpgradeProcessorInternal@1949042653 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@31b182eb:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:55365] msgSent=611 msgReceived=662
>> processor#320: UpgradeProcessorInternal@1970768002 socketWrapper
>> state: Closed NioChannel msgSent=11 msgReceived=12
>> processor#321: UpgradeProcessorInternal@1914450919 socketWrapper
>> state:
>> org.apache.tomcat.util.net.SecureNioChannel@506ac11a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53360] msgSent=19626 msgReceived=550
>> processor#322: UpgradeProcessorInternal@553558786 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#323: UpgradeProcessorInternal@648152412 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=9
>> processor#324: UpgradeProcessorInternal@157417119 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=13
>> processor#325: UpgradeProcessorInternal@794632354 socketWrapper state:
>> Closed NioChannel msgSent=10 msgReceived=12
>> Nuof not connected: 291
>> Nuof connected: 34
>>
>>
>> webSocketMessageBrokerStats: WebSocketSession[33 current
>> WS(33)-HttpStream(0)-HttpPoll(0), 2177 total, 0 closed abnormally (0
>> connect failure, 0 send limit, 251 transport error)]
>> stompSubProtocol[processed CONNECT(2116)-CONNECTED(2116)-DISCONNECT(0)]
>> stompBrokerRelay[null]
>> inboundChannel[pool size = 64, active threads = 0, queued tasks = 0,
>> completed tasks = 542280]
>> outboundChannel[pool size = 64, active threads = 0, queued tasks =
>> 0, completed tasks = 2183026]
>> sockJsScheduler[pool size = 32, active threads = 0, queued tasks =
>> 9, completed tasks = 19964]
>> subProtocolWebSocketHandler:
>> SubProtocolWebSocketHandler[StompSubProtocolHandler[v10.stomp,
>> v11.stomp, v12.stomp]]
>> subProtocolWebSocketHandler.sessions: 33
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Possible AbstractProtocol.waitingProcessors leak in Tomcat 9.0.75
Posted by Mark Thomas <ma...@apache.org>.
Hi Mario,
That does look like a possible bug.
I'll try and do a code review before the next release but from
experience f you are able to figure out how to reproduce it that would
help a lot.
Thanks,
Mark
On 06/07/2023 15:19, mario@datenwort.at.INVALID wrote:
> Hello!
>
> I guess I found a memory leak which I am not quite sure how to reproduce
> yet.
> Our application is using WebSockets and sometimes it seems that Tomcat
> 9.0.75 does not always remove the „UpgradeProcessorInternal“ process
> from the list of „waitingProcessors“ in
> „org.apache.coyote.AbstractProtocol“.
>
> I managed to create a dump (in that e-mail below) of that data from a
> live VM and found that some of those UpgradeProcessorInternal instances
> are referencing a "Closed Socket", which somehow feels odd to me. This
> list slowly grows. The entries are never removed.
> Those UpgradeProcessorInternal-s referencing a live socket are matching
> exactly the number of live WebSocket sessions.
>
> What do you think?
>
> Env:
> Tomcat: 9.0.75.0
> Spring Boot: 2.7.12
> Java: 17.0.7
>
>
> Best regards,
> Mario
>
>
> connector :Connector[HTTP/1.1-2888]
> waitingProcessors: 325
> processor#1: UpgradeProcessorInternal@1087825622 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#2: UpgradeProcessorInternal@92090237 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#3: UpgradeProcessorInternal@1641815002 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#4: UpgradeProcessorInternal@1749164616 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#5: UpgradeProcessorInternal@508142663 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#6: UpgradeProcessorInternal@1482593872 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#7: UpgradeProcessorInternal@1176145940 socketWrapper state:
> Closed NioChannel msgSent=4389 msgReceived=205
> processor#8: UpgradeProcessorInternal@1420482234 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#9: UpgradeProcessorInternal@31846900 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#10: UpgradeProcessorInternal@1726295282 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#11: UpgradeProcessorInternal@288536869 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#12: UpgradeProcessorInternal@969872343 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#13: UpgradeProcessorInternal@420227862 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#14: UpgradeProcessorInternal@552002299 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#15: UpgradeProcessorInternal@255507237 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#16: UpgradeProcessorInternal@724602640 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#17: UpgradeProcessorInternal@1706491794 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#18: UpgradeProcessorInternal@1925947118 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#19: UpgradeProcessorInternal@2044888516 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#20: UpgradeProcessorInternal@905596380 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#21: UpgradeProcessorInternal@897538387 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@61aa8978:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:40566] msgSent=7277 msgReceived=447
> processor#22: UpgradeProcessorInternal@1070005238 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#23: UpgradeProcessorInternal@34919463 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#24: UpgradeProcessorInternal@565924232 socketWrapper state:
> Closed NioChannel msgSent=12 msgReceived=14
> processor#25: UpgradeProcessorInternal@1146279009 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#26: UpgradeProcessorInternal@1467208007 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#27: UpgradeProcessorInternal@348911347 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#28: UpgradeProcessorInternal@38694518 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#29: UpgradeProcessorInternal@1164156250 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#30: UpgradeProcessorInternal@1675936734 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#31: UpgradeProcessorInternal@441511021 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#32: UpgradeProcessorInternal@59206586 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@5910f8ae:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50444] msgSent=3684 msgReceived=449
> processor#33: UpgradeProcessorInternal@1027164423 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#34: UpgradeProcessorInternal@2101191939 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#35: UpgradeProcessorInternal@1598120319 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#36: UpgradeProcessorInternal@1601686839 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#37: UpgradeProcessorInternal@2131114820 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#38: UpgradeProcessorInternal@1695970133 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#39: UpgradeProcessorInternal@1088759468 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@626eafa5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58972] msgSent=727 msgReceived=868
> processor#40: UpgradeProcessorInternal@332728223 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#41: UpgradeProcessorInternal@221860722 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#42: UpgradeProcessorInternal@773355092 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#43: UpgradeProcessorInternal@826604304 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#44: UpgradeProcessorInternal@267595687 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@6c19b8c1:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50565] msgSent=2968 msgReceived=387
> processor#45: UpgradeProcessorInternal@417381557 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#46: UpgradeProcessorInternal@1626896557 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#47: UpgradeProcessorInternal@1326188889 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#48: UpgradeProcessorInternal@1620503747 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#49: UpgradeProcessorInternal@1588637927 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@1ce5fc5a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:33528] msgSent=5765 msgReceived=442
> processor#50: UpgradeProcessorInternal@148940473 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#51: UpgradeProcessorInternal@915453133 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#52: UpgradeProcessorInternal@369115230 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#53: UpgradeProcessorInternal@405977197 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#54: UpgradeProcessorInternal@47138960 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#55: UpgradeProcessorInternal@178012412 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@13f91fab:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48880] msgSent=5 msgReceived=11
> processor#56: UpgradeProcessorInternal@360143126 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#57: UpgradeProcessorInternal@1196637488 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#58: UpgradeProcessorInternal@1741581739 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@2815c011:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59371] msgSent=8604 msgReceived=853
> processor#59: UpgradeProcessorInternal@1053645483 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#60: UpgradeProcessorInternal@490296158 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#61: UpgradeProcessorInternal@280948950 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#62: UpgradeProcessorInternal@1763242354 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#63: UpgradeProcessorInternal@170476101 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#64: UpgradeProcessorInternal@791654722 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#65: UpgradeProcessorInternal@808226370 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#66: UpgradeProcessorInternal@325893404 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#67: UpgradeProcessorInternal@1205002145 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#68: UpgradeProcessorInternal@105139248 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#69: UpgradeProcessorInternal@835317180 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#70: UpgradeProcessorInternal@1110689861 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@2e9114a0:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57172] msgSent=31182 msgReceived=737
> processor#71: UpgradeProcessorInternal@694149416 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#72: UpgradeProcessorInternal@1917387827 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#73: UpgradeProcessorInternal@1669990384 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#74: UpgradeProcessorInternal@482149574 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#75: UpgradeProcessorInternal@1630942539 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#76: UpgradeProcessorInternal@661587730 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#77: UpgradeProcessorInternal@334804362 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#78: UpgradeProcessorInternal@1847955546 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@f2be4c4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60237] msgSent=537 msgReceived=557
> processor#79: UpgradeProcessorInternal@961085898 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#80: UpgradeProcessorInternal@1015170564 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#81: UpgradeProcessorInternal@1472660801 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#82: UpgradeProcessorInternal@1662742940 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#83: UpgradeProcessorInternal@783736881 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#84: UpgradeProcessorInternal@1033748247 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#85: UpgradeProcessorInternal@1631720904 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#86: UpgradeProcessorInternal@377835540 socketWrapper state:
> Closed NioChannel msgSent=9 msgReceived=9
> processor#87: UpgradeProcessorInternal@208792804 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#88: UpgradeProcessorInternal@27480885 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#89: UpgradeProcessorInternal@6538996 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#90: UpgradeProcessorInternal@2030787475 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#91: UpgradeProcessorInternal@1780923582 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#92: UpgradeProcessorInternal@30596938 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#93: UpgradeProcessorInternal@165266752 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#94: UpgradeProcessorInternal@1937101292 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#95: UpgradeProcessorInternal@854833769 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#96: UpgradeProcessorInternal@1653312022 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#97: UpgradeProcessorInternal@946062077 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#98: UpgradeProcessorInternal@111135744 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#99: UpgradeProcessorInternal@1407359357 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@50247a32:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56550] msgSent=17041 msgReceived=1054
> processor#100: UpgradeProcessorInternal@1688084542 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#101: UpgradeProcessorInternal@1506439018 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#102: UpgradeProcessorInternal@515959904 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#103: UpgradeProcessorInternal@259281365 socketWrapper state:
> Closed NioChannel msgSent=12 msgReceived=12
> processor#104: UpgradeProcessorInternal@364095762 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#105: UpgradeProcessorInternal@307308786 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#106: UpgradeProcessorInternal@1677683546 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#107: UpgradeProcessorInternal@318642779 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#108: UpgradeProcessorInternal@914880558 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#109: UpgradeProcessorInternal@291835852 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#110: UpgradeProcessorInternal@168338082 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#111: UpgradeProcessorInternal@1174282578 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#112: UpgradeProcessorInternal@331106577 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#113: UpgradeProcessorInternal@1617882333 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#114: UpgradeProcessorInternal@542714593 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#115: UpgradeProcessorInternal@1156269649 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#116: UpgradeProcessorInternal@201406651 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#117: UpgradeProcessorInternal@2058862092 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#118: UpgradeProcessorInternal@857502172 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#119: UpgradeProcessorInternal@996422055 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#120: UpgradeProcessorInternal@1815696637 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#121: UpgradeProcessorInternal@736438049 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@7b35f1fe:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:58899] msgSent=799 msgReceived=935
> processor#122: UpgradeProcessorInternal@477398194 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#123: UpgradeProcessorInternal@1219089519 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#124: UpgradeProcessorInternal@1272242963 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#125: UpgradeProcessorInternal@121254386 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#126: UpgradeProcessorInternal@1669364040 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#127: UpgradeProcessorInternal@275174062 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#128: UpgradeProcessorInternal@554430912 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#129: UpgradeProcessorInternal@1187725828 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#130: UpgradeProcessorInternal@1531691908 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#131: UpgradeProcessorInternal@446727798 socketWrapper state:
> Closed NioChannel msgSent=283 msgReceived=259
> processor#132: UpgradeProcessorInternal@362658123 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#133: UpgradeProcessorInternal@330299253 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#134: UpgradeProcessorInternal@1417579684 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#135: UpgradeProcessorInternal@79366241 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#136: UpgradeProcessorInternal@1794180138 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#137: UpgradeProcessorInternal@773525189 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#138: UpgradeProcessorInternal@88744340 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#139: UpgradeProcessorInternal@709557909 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#140: UpgradeProcessorInternal@263285102 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#141: UpgradeProcessorInternal@1223181320 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#142: UpgradeProcessorInternal@1288060966 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@106e2540:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:61373] msgSent=9780 msgReceived=297
> processor#143: UpgradeProcessorInternal@102971590 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#144: UpgradeProcessorInternal@808226506 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#145: UpgradeProcessorInternal@1228942248 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#146: UpgradeProcessorInternal@1214228151 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#147: UpgradeProcessorInternal@550718522 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#148: UpgradeProcessorInternal@546382458 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#149: UpgradeProcessorInternal@430113102 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#150: UpgradeProcessorInternal@1801235373 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#151: UpgradeProcessorInternal@613091448 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#152: UpgradeProcessorInternal@567610662 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@7d797520:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:32782] msgSent=7435 msgReceived=471
> processor#153: UpgradeProcessorInternal@864215926 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#154: UpgradeProcessorInternal@257132453 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#155: UpgradeProcessorInternal@1679461613 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#156: UpgradeProcessorInternal@1792036392 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#157: UpgradeProcessorInternal@1237165381 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#158: UpgradeProcessorInternal@2095009319 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#159: UpgradeProcessorInternal@405187807 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#160: UpgradeProcessorInternal@1137415986 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#161: UpgradeProcessorInternal@1259367401 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#162: UpgradeProcessorInternal@590114774 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#163: UpgradeProcessorInternal@1979332358 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#164: UpgradeProcessorInternal@1910582044 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#165: UpgradeProcessorInternal@1205302056 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#166: UpgradeProcessorInternal@1454816183 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#167: UpgradeProcessorInternal@1127766585 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#168: UpgradeProcessorInternal@1563629106 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@2cdc91f6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59195] msgSent=832 msgReceived=865
> processor#169: UpgradeProcessorInternal@27650727 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#170: UpgradeProcessorInternal@1847115547 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#171: UpgradeProcessorInternal@1939276436 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#172: UpgradeProcessorInternal@2038944386 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#173: UpgradeProcessorInternal@1155354584 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#174: UpgradeProcessorInternal@97303750 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#175: UpgradeProcessorInternal@1608676584 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#176: UpgradeProcessorInternal@886887382 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#177: UpgradeProcessorInternal@620014078 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#178: UpgradeProcessorInternal@551004124 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#179: UpgradeProcessorInternal@2110011592 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#180: UpgradeProcessorInternal@1908346035 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#181: UpgradeProcessorInternal@378148743 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#182: UpgradeProcessorInternal@1156173823 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#183: UpgradeProcessorInternal@64974024 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#184: UpgradeProcessorInternal@1009466172 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#185: UpgradeProcessorInternal@536984857 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#186: UpgradeProcessorInternal@383186893 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#187: UpgradeProcessorInternal@836747459 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#188: UpgradeProcessorInternal@1077816609 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#189: UpgradeProcessorInternal@645247338 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=10
> processor#190: UpgradeProcessorInternal@921863122 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#191: UpgradeProcessorInternal@1763449404 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#192: UpgradeProcessorInternal@464409230 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@519deb3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:36638] msgSent=5311 msgReceived=430
> processor#193: UpgradeProcessorInternal@1278303507 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#194: UpgradeProcessorInternal@1244687124 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#195: UpgradeProcessorInternal@1930251816 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#196: UpgradeProcessorInternal@43494834 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@29f70e25:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:51288] msgSent=7 msgReceived=9
> processor#197: UpgradeProcessorInternal@1612242224 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#198: UpgradeProcessorInternal@1831479299 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#199: UpgradeProcessorInternal@287185460 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#200: UpgradeProcessorInternal@1947436860 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#201: UpgradeProcessorInternal@171168002 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#202: UpgradeProcessorInternal@1436058281 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#203: UpgradeProcessorInternal@1647228700 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#204: UpgradeProcessorInternal@658189327 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#205: UpgradeProcessorInternal@1207405250 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#206: UpgradeProcessorInternal@1206888150 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#207: UpgradeProcessorInternal@11407255 socketWrapper state:
> Closed NioChannel msgSent=512 msgReceived=549
> processor#208: UpgradeProcessorInternal@999984803 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@6887705e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59146] msgSent=719 msgReceived=796
> processor#209: UpgradeProcessorInternal@2079580872 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#210: UpgradeProcessorInternal@26762917 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#211: UpgradeProcessorInternal@653767625 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#212: UpgradeProcessorInternal@685357542 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#213: UpgradeProcessorInternal@2105023035 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#214: UpgradeProcessorInternal@906125638 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#215: UpgradeProcessorInternal@1739573994 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#216: UpgradeProcessorInternal@500242071 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#217: UpgradeProcessorInternal@1067985634 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#218: UpgradeProcessorInternal@1526977615 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@113386a5:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56957] msgSent=11502 msgReceived=275
> processor#219: UpgradeProcessorInternal@1564249713 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#220: UpgradeProcessorInternal@1271579268 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#221: UpgradeProcessorInternal@1251932623 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#222: UpgradeProcessorInternal@336201051 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#223: UpgradeProcessorInternal@1969809982 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#224: UpgradeProcessorInternal@1001859809 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#225: UpgradeProcessorInternal@1073172128 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#226: UpgradeProcessorInternal@787383222 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#227: UpgradeProcessorInternal@599435490 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#228: UpgradeProcessorInternal@202250581 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#229: UpgradeProcessorInternal@2120255293 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#230: UpgradeProcessorInternal@1888709582 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#231: UpgradeProcessorInternal@765967098 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#232: UpgradeProcessorInternal@1517419823 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#233: UpgradeProcessorInternal@1701040641 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#234: UpgradeProcessorInternal@1146615605 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#235: UpgradeProcessorInternal@117512293 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#236: UpgradeProcessorInternal@2012342422 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#237: UpgradeProcessorInternal@1973886667 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#238: UpgradeProcessorInternal@1105514123 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#239: UpgradeProcessorInternal@1632774205 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@2e335b3f:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:60332] msgSent=1124 msgReceived=510
> processor#240: UpgradeProcessorInternal@1643707528 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#241: UpgradeProcessorInternal@1788721645 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#242: UpgradeProcessorInternal@1338930879 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#243: UpgradeProcessorInternal@1437970118 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#244: UpgradeProcessorInternal@825036894 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#245: UpgradeProcessorInternal@1411771222 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#246: UpgradeProcessorInternal@1641563307 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#247: UpgradeProcessorInternal@1910767760 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#248: UpgradeProcessorInternal@1376679291 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#249: UpgradeProcessorInternal@1369067244 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#250: UpgradeProcessorInternal@640417107 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@12ac4472:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:56706] msgSent=725 msgReceived=838
> processor#251: UpgradeProcessorInternal@1723934650 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#252: UpgradeProcessorInternal@390093373 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#253: UpgradeProcessorInternal@1679102358 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#254: UpgradeProcessorInternal@1943695966 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#255: UpgradeProcessorInternal@1675844197 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@46028bc6:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59060] msgSent=1799 msgReceived=1047
> processor#256: UpgradeProcessorInternal@1597795515 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#257: UpgradeProcessorInternal@449048399 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#258: UpgradeProcessorInternal@1138357335 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#259: UpgradeProcessorInternal@1150701318 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#260: UpgradeProcessorInternal@759289043 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#261: UpgradeProcessorInternal@663090198 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#262: UpgradeProcessorInternal@1804724738 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#263: UpgradeProcessorInternal@1681171367 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#264: UpgradeProcessorInternal@894140638 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#265: UpgradeProcessorInternal@1268650502 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#266: UpgradeProcessorInternal@201391001 socketWrapper state:
> Closed NioChannel msgSent=9 msgReceived=9
> processor#267: UpgradeProcessorInternal@4454872 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@deddb3c:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50534] msgSent=2955 msgReceived=422
> processor#268: UpgradeProcessorInternal@1292060318 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#269: UpgradeProcessorInternal@1799841497 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#270: UpgradeProcessorInternal@1293709 socketWrapper state:
> Closed NioChannel msgSent=9 msgReceived=9
> processor#271: UpgradeProcessorInternal@1454181134 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#272: UpgradeProcessorInternal@1434636325 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#273: UpgradeProcessorInternal@1318115128 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@6c092b03:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:42084] msgSent=17154 msgReceived=980
> processor#274: UpgradeProcessorInternal@63391341 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#275: UpgradeProcessorInternal@1941813270 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@1202bac8:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50836] msgSent=8979 msgReceived=1442
> processor#276: UpgradeProcessorInternal@1709007472 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#277: UpgradeProcessorInternal@1687754548 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#278: UpgradeProcessorInternal@1353840384 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#279: UpgradeProcessorInternal@689966738 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#280: UpgradeProcessorInternal@769715798 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@71f5b17e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:48881] msgSent=5 msgReceived=8
> processor#281: UpgradeProcessorInternal@538693548 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#282: UpgradeProcessorInternal@99010652 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#283: UpgradeProcessorInternal@46163835 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#284: UpgradeProcessorInternal@1653861160 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#285: UpgradeProcessorInternal@835521137 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#286: UpgradeProcessorInternal@1819046354 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#287: UpgradeProcessorInternal@704460355 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@1329f2a4:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:57088] msgSent=16877 msgReceived=962
> processor#288: UpgradeProcessorInternal@999693398 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#289: UpgradeProcessorInternal@952037244 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#290: UpgradeProcessorInternal@1834838687 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#291: UpgradeProcessorInternal@1353907057 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#292: UpgradeProcessorInternal@1920282040 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#293: UpgradeProcessorInternal@502347326 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#294: UpgradeProcessorInternal@521254110 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#295: UpgradeProcessorInternal@642772382 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#296: UpgradeProcessorInternal@785858310 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=13
> processor#297: UpgradeProcessorInternal@1208022481 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@781a5539:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53273] msgSent=1748 msgReceived=1047
> processor#298: UpgradeProcessorInternal@2131665626 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@3b699188:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50660] msgSent=275 msgReceived=428
> processor#299: UpgradeProcessorInternal@969235472 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#300: UpgradeProcessorInternal@632167035 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@138c331e:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:59302] msgSent=1638 msgReceived=830
> processor#301: UpgradeProcessorInternal@344672093 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#302: UpgradeProcessorInternal@444927827 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#303: UpgradeProcessorInternal@826205929 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#304: UpgradeProcessorInternal@8738130 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#305: UpgradeProcessorInternal@1324275001 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@3dc58c00:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:52942] msgSent=48 msgReceived=13
> processor#306: UpgradeProcessorInternal@1327508728 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> processor#307: UpgradeProcessorInternal@1235248762 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#308: UpgradeProcessorInternal@757308671 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#309: UpgradeProcessorInternal@394008226 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#310: UpgradeProcessorInternal@1948325825 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#311: UpgradeProcessorInternal@1802817168 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#312: UpgradeProcessorInternal@621216482 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#313: UpgradeProcessorInternal@1786662299 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@377ebd56:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:49470] msgSent=810 msgReceived=966
> processor#314: UpgradeProcessorInternal@1842470969 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@2bc4e23a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:50677] msgSent=809 msgReceived=953
> processor#315: UpgradeProcessorInternal@1885340041 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#316: UpgradeProcessorInternal@836220989 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#317: UpgradeProcessorInternal@446331766 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=9
> processor#318: UpgradeProcessorInternal@352928484 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#319: UpgradeProcessorInternal@1949042653 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@31b182eb:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:55365] msgSent=611 msgReceived=662
> processor#320: UpgradeProcessorInternal@1970768002 socketWrapper state:
> Closed NioChannel msgSent=11 msgReceived=12
> processor#321: UpgradeProcessorInternal@1914450919 socketWrapper state:
> org.apache.tomcat.util.net.SecureNioChannel@506ac11a:java.nio.channels.SocketChannel[connected local=/***.***.***.***:2888 remote=/***.***.***.***:53360] msgSent=19626 msgReceived=550
> processor#322: UpgradeProcessorInternal@553558786 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#323: UpgradeProcessorInternal@648152412 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=9
> processor#324: UpgradeProcessorInternal@157417119 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=13
> processor#325: UpgradeProcessorInternal@794632354 socketWrapper state:
> Closed NioChannel msgSent=10 msgReceived=12
> Nuof not connected: 291
> Nuof connected: 34
>
>
> webSocketMessageBrokerStats: WebSocketSession[33 current
> WS(33)-HttpStream(0)-HttpPoll(0), 2177 total, 0 closed abnormally (0
> connect failure, 0 send limit, 251 transport error)]
> stompSubProtocol[processed CONNECT(2116)-CONNECTED(2116)-DISCONNECT(0)]
> stompBrokerRelay[null]
> inboundChannel[pool size = 64, active threads = 0, queued tasks = 0,
> completed tasks = 542280]
> outboundChannel[pool size = 64, active threads = 0, queued tasks = 0,
> completed tasks = 2183026]
> sockJsScheduler[pool size = 32, active threads = 0, queued tasks = 9,
> completed tasks = 19964]
> subProtocolWebSocketHandler:
> SubProtocolWebSocketHandler[StompSubProtocolHandler[v10.stomp,
> v11.stomp, v12.stomp]]
> subProtocolWebSocketHandler.sessions: 33
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org