You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Phil Sorber (JIRA)" <ji...@apache.org> on 2016/11/02 16:55:59 UTC
[jira] [Commented] (TS-4911) ATS restarts when ts_lua plugin is
used with collapsed_forwarding plugin and write lock error was encountered
(thundering herd problem)
[ https://issues.apache.org/jira/browse/TS-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15629576#comment-15629576 ]
Phil Sorber commented on TS-4911:
---------------------------------
[~kichan], [~rkishore], Can one of you create a back port PR for this following the method I outlined at the Summit? Thanks.
> ATS restarts when ts_lua plugin is used with collapsed_forwarding plugin and write lock error was encountered (thundering herd problem)
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: TS-4911
> URL: https://issues.apache.org/jira/browse/TS-4911
> Project: Traffic Server
> Issue Type: Bug
> Components: Lua, Plugins
> Affects Versions: 6.2.1, 7.0.0, 7.1.0
> Reporter: Rajendra Kishore Bonumahanti
> Assignee: Kit Chan
> Fix For: 7.0.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> ATS (ts_lua plugin is used along with collapsed_forwarding plugin) got restarted due to “FATAL: InkAPI.cc:3800: failed assert `sdk_sanity_check_http_hdr_handle(obj) == TS_SUCCESS`” when write lock error was encountered (thundering herd problem). The ATS restart is due to ts.client_response.get_status() function call in TS_LUA_HOOK_SEND_RESPONSE_HDR hook function when re-direct is enabled on 502 error.
> == error when ts_lua plugin is used with collapsed_forwarding plugin ==
> map http://testurl.com/ http://origin.com/ @plugin=tslua.so @pparam=/opt/trafficserver/etc/trafficserver/lua/test.lua @plugin=collapsed_forwarding.so @pparam=--delay=10 @pparam=--retries=5
> test.lua:
> =======
> function __init__()
> local FUNCTION = '__init__(): '
> ts.debug(FUNCTION .. 'Entering..')
> return 0
> end
> function responseHook()
> local FUNCTION = 'responseHook(): '
> ts.debug(FUNCTION .. 'Entering..')
> local c_code = tostring(ts.client_response.get_status())
> ts.debug(FUNCTION .. 'Original client_response_code: ' .. c_code)
> if string.find(c_code, "^[45]") then
> ts.debug(FUNCTION .. 'client response code is now set to 500')
> ts.client_response.set_status(500)
> end
> return 0
> end
> function do_remap()
> local FUNCTION = 'do_remap(): '
> ts.debug(FUNCTION .. 'Entering..')
> ts.hook(TS_LUA_HOOK_SEND_RESPONSE_HDR, responseHook)
> return 0
> end
> ========================================================
> The log…
> [Sep 20 19:53:43.746] Server {0x2ad69673e700} DEBUG: (http) [148] State Transition: SM_ACTION_AP_CACHE_LOOKUP_COMPLETE -> SM_ACTION_SERVE_FROM_CACHE
> [Sep 20 19:53:43.746] Server {0x2ad69673e700} DEBUG: (http) [148] perform_cache_write_action CACE_DO_SERVE
> [Sep 20 19:53:43.746] Server {0x2ad69673e700} DEBUG: (http_redirect) is_redirect_required 1
> [Sep 20 19:53:43.746] Server {0x2ad69673e700} DEBUG: (http) [148] calling plugin on hook TS_HTTPSEND_RESPONSE_HDR_HOOK at hook 0x1832bc0
> [Sep 20 19:53:43.746] Server {0x2ad69673e700} DIAG: (ts_lua) responseHook(): Entering..
> FATAL: InkAPI.cc:3800: failed assert `sdk_sanity_check_http_hdr_handle(obj) == TS_SUCCESS`
> traffic_server: using root directory '/opt/trafficserver'
> validate_filter_args: "action=deny"
> -----------------------------------------------------------------------------------------
> Filter "<NONAME>" status: allow_flag=false, src_ip_valid=false, in_ip_valid=false, internal=false active_queue_flag=0
> standard methods=nonstandard methods=
> src_ip_cnt=0
> in_ip_cnt=0
> URL Rewrite table with 3 entries
> Reverse Proxy is On
> Forward Mapping Table with 3 entries
> http:/// => http://127.0.0.1/ <> [plugins not enabled; running with 0 plugins]
> http://xx.xx.xx.xx/ => http://xx.xx.xx.xx/ <> [plugins are enabled; running with plugins]
> http://testurl.com/ => http://origin.com// <> [plugins are enabled; running with 2 plugins]
> Reverse Mapping Table with 0 entries
> Permanent Redirect Mapping Table with 0 entries
> Temporary Redirect Mapping Table with 0 entries
> Forward Mapping With Recv Port Table with 0 entries
> Referer filter default redirect URL: "http://www.example.com/"
> traffic_server: Aborted (Signal sent by tkill() 129399 99)
> traffic_server - STACK TRACE:
> /opt/trafficserver/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4ab5ee]
> /lib64/libpthread.so.0(+0xf100)[0x2ad67f3eb100]
> /lib64/libc.so.6(gsignal+0x37)[0x2ad6803bb5f7]
> /lib64/libc.so.6(abort+0x148)[0x2ad6803bcce8]
> /opt/trafficserver/lib/libtsutil.so.6(+0x28668)[0x2ad67d927668]
> /opt/trafficserver/lib/libtsutil.so.6(+0x286fc)[0x2ad67d9276fc]
> /opt/trafficserver/lib/libtsutil.so.6(+0x269e5)[0x2ad67d9259e5]
> /opt/trafficserver/bin/traffic_server[0x4c7973]
> /opt/trafficserver/libexec/trafficserver/tslua.so(+0x9aeb)[0x2ad695821aeb]
> /opt/trafficserver/bin/traffic_server[0x5834e6]
> ========================================================
> It appears, the following code in ts_lua [ts_lua_client_response.c] plugin causes the sdk_sanity check failed.
> #define TS_LUA_CHECK_CLIENT_RESPONSE_HDR(http_ctx) \
> do { \
> if (!http_ctx->client_response_hdrp) { \
> if (TSHttpTxnClientRespGet(http_ctx->txnp, &http_ctx->client_response_bufp, &http_ctx->client_response_hdrp) != \
> TS_SUCCESS) { \
> return 0; \
> } \
> } \
> } while (0)
> ….
> ….
> static int
> ts_lua_client_response_get_status(lua_State *L)
> {
> int status;
> ts_lua_http_ctx *http_ctx;
> GET_HTTP_CONTEXT(http_ctx, L);
> TS_LUA_CHECK_CLIENT_RESPONSE_HDR(http_ctx);
> status = TSHttpHdrStatusGet(http_ctx->client_response_bufp, http_ctx->client_response_hdrp);
> lua_pushinteger(L, status);
> return 1;
> }
> ….
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)