You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by shinrich <gi...@git.apache.org> on 2017/03/09 20:24:41 UTC
[GitHub] trafficserver issue #1561: A new 7.1 Crash
GitHub user shinrich opened an issue:
https://github.com/apache/trafficserver/issues/1561
A new 7.1 Crash
After running in production for 2.5 days without incident (with a couple fixes now merged back to 7.1.x), I got a core with the following stack trace
```
(gdb) bt
#0 0x00002ba487c1c625 in raise () from /lib64/libc.so.6
#1 0x00002ba487c1dd8d in abort () from /lib64/libc.so.6
#2 0x00002ba48576b5a9 in ink_abort (message_format=0x2ba48577d75c "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3 0x00002ba485768cda in _ink_assert (expression=0x7b3d70 "((INKContInternal *)contp)->mutex", file=0x7b2dad "InkAPI.cc", line=4372)
at ink_assert.cc:37
#4 0x000000000052bab7 in _TSReleaseAssert (text=0x7b3d70 "((INKContInternal *)contp)->mutex", file=0x7b2dad "InkAPI.cc", line=4372)
at InkAPI.cc:401
#5 0x0000000000534f77 in TSContSchedule (contp=0x2aad1801bd60, timeout=300000, tp=TS_THREAD_POOL_TASK) at InkAPI.cc:4372
#6 0x00002aaabbc39132 in config_handler (cont=0x1fe9ca0, event=Unhandled dwarf expression opcode 0xf3
) at regex_revalidate/regex_revalidate.c:371
#7 0x000000000052c9fb in INKContInternal::handle_event (this=0x1fe9ca0, event=2, edata=0x2aad10079b80) at InkAPI.cc:1049
#8 0x00000000005184f0 in Continuation::handleEvent (this=0x1fe9ca0, event=2, data=0x2aad10079b80)
at /home/shinrich/yats_build/trafficserver/iocore/eventsystem/I_Continuation.h:153
#9 0x00000000007aac3f in EThread::process_event (this=0x109abc90, e=0x2aad10079b80, calling_code=2) at UnixEThread.cc:143
#10 0x00000000007aaf57 in EThread::execute (this=0x109abc90) at UnixEThread.cc:225
#11 0x00000000007aa303 in spawn_thread_internal (a=0x105b0280) at Thread.cc:84
#12 0x00002ba487f86aa1 in start_thread () from /lib64/libpthread.so.0
#13 0x00002ba487cd293d in clone () from /lib64/libc.so.6
(gdb)
```
The problem is that TSContSchedule calls FORCE_PLUGIN_SCOPED_MUTEX which asserts that the continuation has a mutex. In this case, the continuation does not have a mutex. This check was added last May to solve TS-4387.
The offending plugin is regex_revalidate. If the configuration file is updated, it creates a continuation and spins it off on a task thread. But it is calling TSContCreate with a second argument of NULL. Indeed the timestamp of the config file is a minute before the timestamp on the core file.
The straightforward fix would be create a mutex to pass into TSContCreate. Not sure what the downside of that would be. It appears that this is not a frequently run path, so I doubt the creation of an additional object would affect much one way or the other.
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1561: A new 7.1 Crash with regex_revalidate config upda...
Posted by zwoop <gi...@git.apache.org>.
Github user zwoop commented on the issue:
https://github.com/apache/trafficserver/issues/1561
Yeh, I think you have to create the mutex. Before all these changes, the core would create one for you if it needed it, but that had a massive single mutex around that. So instead, we now assert, and the plugins must be updated, because they are using APIs that requires the mutex.
It is possible there's a bug somewhere, but odds are that the plugin is using an API that needs a cont mutex.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] trafficserver issue #1561: A new 7.1 Crash with regex_revalidate config upda...
Posted by shinrich <gi...@git.apache.org>.
Github user shinrich commented on the issue:
https://github.com/apache/trafficserver/issues/1561
@jrushford fixed this for 6.2.x. I'm putting up a PR with a cherry-pick of his fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---