You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/11/11 02:53:16 UTC
[GitHub] [trafficserver] masaori335 opened a new issue #7315: 8.1.0 Crash on write_dns
masaori335 opened a new issue #7315:
URL: https://github.com/apache/trafficserver/issues/7315
We faced crashes on v8.1.0 when the DNS server gets slow.
```
[ 0 ] libpthread-2.17.so waitpid ( :undefined )
[ 1 ] traffic_server crash_logger_invoke(int, siginfo_t*, void*) ( Crash.cc:165 )
[ 2 ] libpthread-2.17.so 0x7fbfcaebd630 ( :undefined )
[ 3 ] libtscore.so.8.1.0 ink_atomiclist_push ( ink_queue.cc:533 )
[ 4 ] traffic_server ProtectedQueue::enqueue(Event*, bool) ( ProtectedQueue.cc:52 )
[ 5 ] traffic_server write_dns(DNSHandler*, bool) ( DNS.cc:undefined )
[ 6 ] traffic_server DNSHandler::mainEvent(int, Event*) ( DNS.cc:956 )
[ 7 ] traffic_server NetHandler::waitForActivity(long) ( UnixNet.cc:519 )
[ 8 ] traffic_server non-virtual thunk to NetHandler::waitForActivity(long) ( UnixNet.cc:undefined )
[ 9 ] traffic_server EThread::execute_regular() ( UnixEThread.cc:278 )
[ 10 ] traffic_server spawn_thread_internal(void*) ( Thread.cc:85 )
[ 11 ] libpthread-2.17.so start_thread ( :undefined )
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on issue #7315: 8.1.0 Crash on write_dns
Posted by GitBox <gi...@apache.org>.
masaori335 commented on issue #7315:
URL: https://github.com/apache/trafficserver/issues/7315#issuecomment-726580203
I'm pretty sure this happens with SplitDNS. The `DNSHandler` mutex of `SplitDNS` is not locked.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on issue #7315: 8.1.0 Crash on write_dns
Posted by GitBox <gi...@apache.org>.
masaori335 commented on issue #7315:
URL: https://github.com/apache/trafficserver/issues/7315#issuecomment-725097360
```
(gdb) bt
#0 0x00007fe90c4e61d9 in waitpid () from /lib64/libpthread.so.0
#1 0x000055c6a88f8024 in crash_logger_invoke(int, siginfo_t*, void*) (signo=11, info=0x7fe8ff5090b0,
ctx=0x7fe8ff508f80) at traffic_server/Crash.cc:165
#2 <signal handler called>
#3 ink_atomiclist_push (l=0x100b10, item=0x7fe90a9e4d80) at ink_queue.cc:533
#4 0x000055c6a8b03f83 in ProtectedQueue::enqueue(Event*, bool) (this=0x100b10, e=0x7fe90a9e4d80,
fast_signal=false) at ProtectedQueue.cc:52
#5 0x000055c6a8a46224 in write_dns_event (h=0x7fe90674ec00, e=0x7fe8ff302b00, over_tcp=<optimized out>)
at /usr/src/debug/bazinga-trafficserver-8.1.0/iocore/eventsystem/P_UnixEvent.h:30
#6 write_dns(DNSHandler*, bool) (h=0x7fe90674ec00, tcp_retry=<optimized out>) at DNS.cc:1038
#7 0x000055c6a8a42717 in DNSHandler::mainEvent(int, Event*) (this=0x7fe90674ec00,
event=<optimized out>, e=<optimized out>) at DNS.cc:956
#8 0x000055c6a8ad86fe in NetHandler::waitForActivity(long) (this=<optimized out>,
timeout=<optimized out>) at UnixNet.cc:519
#9 0x000055c6a8ad8bfd in non-virtual thunk to NetHandler::waitForActivity(long) ()
#10 0x000055c6a8b05d81 in EThread::execute_regular() (this=0x7fe8ff60dc40) at UnixEThread.cc:278
#11 0x000055c6a8b04b0c in spawn_thread_internal(void*) (a=0x7fe90a995320) at Thread.cc:85
#12 0x00007fe90c4deea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fe90b8fc8dd in clone () from /lib64/libc.so.6
```
It looks like `ProtectedQueue` on frame 4 is pointing to an invalid address.
```
(gdb) frame 4
#4 0x000055c6a8b03f83 in ProtectedQueue::enqueue(Event*, bool) (this=0x100b10, e=0x7fe90a9e4d80,
fast_signal=false) at ProtectedQueue.cc:52
52 bool was_empty = (ink_atomiclist_push(&al, e) == nullptr);
(gdb) p this
$12 = (struct ProtectedQueue *) 0x100b10
```
This might be made by referring `thread_holding` of unlocked mutex on `write_dns_event`.
```
(gdb) frame 6
#6 write_dns(DNSHandler*, bool) (h=0x7fe90674ec00, tcp_retry=<optimized out>) at DNS.cc:1038
1038 if (h->ns_down[h->name_server] || !write_dns_event(h, e, over_tcp)) {
(gdb) p *h->mutex.m_ptr
$13 = {<RefCountObj> = {<> = {_vptr$ForceVFPTToTop = 0x55c6a8dc1e78 <vtable for ProxyMutex+16>},
m_refcount = 8}, the_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, thread_holding = 0x0, nthread_holding = 0}
```
https://github.com/apache/trafficserver/blob/c080cc1ec39e21b04ba28e76a6ad80927b71f798/iocore/dns/DNS.cc#L1144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 closed issue #7315: 8.1.0 Crash on write_dns
Posted by GitBox <gi...@apache.org>.
masaori335 closed issue #7315:
URL: https://github.com/apache/trafficserver/issues/7315
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 edited a comment on issue #7315: 8.1.0 Crash on write_dns
Posted by GitBox <gi...@apache.org>.
masaori335 edited a comment on issue #7315:
URL: https://github.com/apache/trafficserver/issues/7315#issuecomment-725097360
```
(gdb) bt
#0 0x00007fe90c4e61d9 in waitpid () from /lib64/libpthread.so.0
#1 0x000055c6a88f8024 in crash_logger_invoke(int, siginfo_t*, void*) (signo=11, info=0x7fe8ff5090b0,
ctx=0x7fe8ff508f80) at traffic_server/Crash.cc:165
#2 <signal handler called>
#3 ink_atomiclist_push (l=0x100b10, item=0x7fe90a9e4d80) at ink_queue.cc:533
#4 0x000055c6a8b03f83 in ProtectedQueue::enqueue(Event*, bool) (this=0x100b10, e=0x7fe90a9e4d80,
fast_signal=false) at ProtectedQueue.cc:52
#5 0x000055c6a8a46224 in write_dns_event (h=0x7fe90674ec00, e=0x7fe8ff302b00, over_tcp=<optimized out>)
at P_UnixEvent.h:30
#6 write_dns(DNSHandler*, bool) (h=0x7fe90674ec00, tcp_retry=<optimized out>) at DNS.cc:1038
#7 0x000055c6a8a42717 in DNSHandler::mainEvent(int, Event*) (this=0x7fe90674ec00,
event=<optimized out>, e=<optimized out>) at DNS.cc:956
#8 0x000055c6a8ad86fe in NetHandler::waitForActivity(long) (this=<optimized out>,
timeout=<optimized out>) at UnixNet.cc:519
#9 0x000055c6a8ad8bfd in non-virtual thunk to NetHandler::waitForActivity(long) ()
#10 0x000055c6a8b05d81 in EThread::execute_regular() (this=0x7fe8ff60dc40) at UnixEThread.cc:278
#11 0x000055c6a8b04b0c in spawn_thread_internal(void*) (a=0x7fe90a995320) at Thread.cc:85
#12 0x00007fe90c4deea5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007fe90b8fc8dd in clone () from /lib64/libc.so.6
```
It looks like `ProtectedQueue` on frame 4 is pointing to an invalid address.
```
(gdb) frame 4
#4 0x000055c6a8b03f83 in ProtectedQueue::enqueue(Event*, bool) (this=0x100b10, e=0x7fe90a9e4d80,
fast_signal=false) at ProtectedQueue.cc:52
52 bool was_empty = (ink_atomiclist_push(&al, e) == nullptr);
(gdb) p this
$12 = (struct ProtectedQueue *) 0x100b10
```
This might be made by referring `thread_holding` of unlocked mutex on `write_dns_event`.
```
(gdb) frame 6
#6 write_dns(DNSHandler*, bool) (h=0x7fe90674ec00, tcp_retry=<optimized out>) at DNS.cc:1038
1038 if (h->ns_down[h->name_server] || !write_dns_event(h, e, over_tcp)) {
(gdb) p *h->mutex.m_ptr
$13 = {<RefCountObj> = {<> = {_vptr$ForceVFPTToTop = 0x55c6a8dc1e78 <vtable for ProxyMutex+16>},
m_refcount = 8}, the_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0,
__kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, thread_holding = 0x0, nthread_holding = 0}
```
https://github.com/apache/trafficserver/blob/c080cc1ec39e21b04ba28e76a6ad80927b71f798/iocore/dns/DNS.cc#L1144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org