You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "kenballus (via GitHub)" <gi...@apache.org> on 2023/07/31 16:36:55 UTC

[GitHub] [trafficserver] kenballus opened a new issue, #10116: Segmentation Fault with `attach_server_session_to_client` and Puma backend

kenballus opened a new issue, #10116:
URL: https://github.com/apache/trafficserver/issues/10116

   # Description
   ATS, when configured with `attach_server_session_to_client`, segfaults when forwarding a request to a Puma backend.
   
   # Steps to reproduce
   0. Start a fresh Debian system.
   ```bash
   docker --workdir /repro -it debian:bookworm
   ```
   1. Install ATS dependencies.
   ```bash
   apt -y update && apt -y upgrade && apt -y install make autoconf automake libtool pkg-config gcc g++ zlib1g-dev libssl-dev libpcre3-dev libcap-dev libhwloc-dev libncurses5-dev libcurl4-openssl-dev flex libunwind-dev git
   ```
   2. Build and install ATS `master`. (The current commit at time of writing is `e6182d9ac9c3f611cb33b3ef6dc98327df41c3d6`)
   ```bash
   git clone "https://github.com/apache/trafficserver" && cd trafficserver && autoreconf -if && ./configure --enable-debug && make -j$(nproc) && make install
   ```
   3. Install Puma dependencies.
   ```bash
   apt -y install ruby-dev && gem install sinatra --version 3.0.6
   ```
   4. Install Puma.
   ```bash
   gem install puma --version 6.3.0
   ```
   5. Copy the files in the "Files" section (below) into the filesystem.
   6. Start the Puma server.
   ```bash
   ruby /repro/server.rb &
   ```
   7. Start ATS.
   ```bash
   traffic_server &
   ```
   8. Install netcat.
   ```bash
   apt -y install netcat-traditional
   ```
   9. Use netcat to send ATS an HTTP request with a short client-side timeout.
   ```bash
   printf 'GET / HTTP/1.1\r\n\r\n' | nc -q 1 localhost 80
   ```
   10. Observe that ATS segfaults and crashes with the following output:
   ```
   [Jul 31 16:31:00.644] traffic_crashlo NOTE: crashlog started, target=35696, debug=false syslog=true, uid=65534 euid=0
   [Jul 31 16:31:00.645] traffic_crashlo NOTE: logging to 0x564faa1de830
   [Jul 31 16:31:00.645] traffic_crashlo NOTE: readlink failed with Permission denied
   [Jul 31 16:31:00.645] traffic_crashlo ERROR: wrote crash log to /usr/local/var/log/trafficserver/crash-2023-07-31-163100.log
   traffic_server: received signal 11 (Segmentation fault)
   traffic_server - STACK TRACE:
   traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0xc5)[0x564195bcfecb]
   /lib/x86_64-linux-gnu/libc.so.6(+0x3bfd0)[0x7fda33fd6fd0]
   traffic_server(_ZN18Http1ClientSession11do_io_closeEi+0xce)[0x564195c7a1a4]
   traffic_server(_ZN18Http1ClientSession16state_keep_aliveEiPv+0x3b4)[0x564195c7b424]
   traffic_server(_ZN12Continuation11handleEventEiPv+0xe7)[0x564195bd96f3]
   traffic_server(+0x97da01)[0x564196051a01]
   traffic_server(+0x97e631)[0x564196052631]
   traffic_server(+0x97ed0c)[0x564196052d0c]
   traffic_server(_ZN18UnixNetVConnection11net_read_ioEP10NetHandlerP7EThread+0x2b)[0x5641960556b9]
   traffic_server(_ZN10NetHandler18process_ready_listEv+0x9b)[0x564196088793]
   traffic_server(_ZN10NetHandler15waitForActivityEl+0x16d)[0x564196088b47]
   traffic_server(_ZN7EThread15execute_regularEv+0x499)[0x5641960c8a95]
   traffic_server(_ZN7EThread7executeEv+0x10b)[0x5641960c8c3f]
   traffic_server(+0x9f341c)[0x5641960c741c]
   /lib/x86_64-linux-gnu/libc.so.6(+0x89044)[0x7fda34024044]
   /lib/x86_64-linux-gnu/libc.so.6(__clone+0x40)[0x7fda340a3860]
   [2]+  Segmentation fault      (core dumped) traffic_server
   ```
   
   # Files
   ## `/usr/local/etc/trafficserver/records.yaml`
   ```yaml
   ts:
     http:
       server_ports: 80 80:ipv6
       attach_server_session_to_client: 1
   ```
   ## `/usr/local/etc/trafficserver/remap.config`
   ```
   map / http://127.0.0.1:8000
   ```
   ## `/repro/server.rb`
   ```ruby
   require 'sinatra/base'
   require 'rack/handler/puma'
   
   class App < Sinatra::Base
     get '*' do
       ""
     end
   end
   
   Rack::Handler::Puma.run(App.new, Port: 8000)
   ```
   
   # Crash log
   ```
   Process:            [TS_MAIN] [35696]
   Version:            Traffic Server 10.0.0
   System Version:     Linux x86_64 #1 SMP PREEMPT_DYNAMIC Sat, 15 Jul 2023 19:25:49 +0000 6.4.3-arch1-2
   Date:               Mon, 31 Jul 2023 16:31:00 +0000
   
   No target signal information
   
   No target CPU registers
   
   Process Status:
   Name: [TS_MAIN]
   Umask:  0022
   State:  S (sleeping)
   Tgid: 35696
   Ngid: 0
   Pid:  35696
   PPid: 1
   TracerPid:  0
   Uid:  65534 65534 65534 65534
   Gid:  65534 65534 65534 65534
   FDSize: 256
   Groups: 65534
   NStgid: 35696
   NSpid:  35696
   NSpgid: 35696
   NSsid:  1
   Kthread:  0
   VmPeak:  3154692 kB
   VmSize:  3092708 kB
   VmLck:         0 kB
   VmPin:         0 kB
   VmHWM:    112092 kB
   VmRSS:    112092 kB
   RssAnon:     90844 kB
   RssFile:     21248 kB
   RssShmem:        0 kB
   VmData:   150592 kB
   VmStk:       132 kB
   VmExe:      5540 kB
   VmLib:     10508 kB
   VmPTE:       560 kB
   VmSwap:        0 kB
   HugetlbPages:        0 kB
   CoreDumping:  0
   THP_enabled:  1
   untag_mask: 0xffffffffffffffff
   Threads:  48
   SigQ: 0/579165
   SigPnd: 0000000000000000
   ShdPnd: 0000000000000000
   SigBlk: 0000000000000000
   SigIgn: 0000000000001001
   SigCgt: 0000000100004efe
   CapInh: 0000000000000000
   CapPrm: 000000000000040a
   CapEff: 0000000000000400
   CapBnd: 00000000a80425fb
   CapAmb: 0000000000000000
   NoNewPrivs: 0
   Seccomp:  2
   Seccomp_filters:  1
   Speculation_Store_Bypass: thread vulnerable
   SpeculationIndirectBranch:  conditional enabled
   Cpus_allowed: ffffffff
   Cpus_allowed_list:  0-31
   Mems_allowed: 00000003
   Mems_allowed_list:  0-1
   voluntary_ctxt_switches:  41
   nonvoluntary_ctxt_switches: 33
   
   Process Limits:
   Limit                     Soft Limit           Hard Limit           Units
   Max cpu time              unlimited            unlimited            seconds
   Max file size             unlimited            unlimited            bytes
   Max data size             unlimited            unlimited            bytes
   Max stack size            unlimited            unlimited            bytes
   Max core file size        unlimited            unlimited            bytes
   Max resident set          unlimited            unlimited            bytes
   Max processes             unlimited            unlimited            processes
   Max open files            1073741816           1073741816           files
   Max locked memory         8388608              8388608              bytes
   Max address space         unlimited            unlimited            bytes
   Max file locks            unlimited            unlimited            locks
   Max pending signals       579165               579165               signals
   Max msgqueue size         819200               819200               bytes
   Max nice priority         0                    0
   Max realtime priority     0                    0
   Max realtime timeout      unlimited            unlimited            us
   ```
   
   # Versions
   ATS: master branch, commit `e6182d9ac9c3f611cb33b3ef6dc98327df41c3d6`
   OS: Debian Bookworm container running on top of Arch Linux with a 6.4.3 kernel
   All other versions are either Debian defaults or specified in the reproduction steps.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] cmcfarlen commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1721081390

   @kenballus Now that this is merged, could you please verify this fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] shinrich commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "shinrich (via GitHub)" <gi...@apache.org>.
shinrich commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1714700863

   Chris noted that this may be the same error I'm seeing https://github.com/apache/trafficserver/issues/10396.
   
   I was running with ASAN enabled, so the use-after-free triggered reliably although I wasn't seeing the crash (much I think).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] cmcfarlen commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1716949230

   I reran the reproduction steps after checking out pr #10399 and it no longer crashes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] cmcfarlen commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1672035138

   This crash is due to a use after free that occurs when a client transaction with an attached server session closes *after* the server session is released.  The server session has a state of PoolableSession::KA_RESERVED, but the server session teardown code doesn't handle this so destroys itself even though the client still has a pointer to it.  I will create a PR that fixes the crash, but I haven't yet figured out the appropriate way to get the server session returned to the global pool.  I suspect this bug might be an old one created by #7849 but I don't know the history or code well enough to be certain.  After I create the PR I'll ask for further help to fully resolve this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] cmcfarlen commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1668292551

   Thank you for the very details reproduction steps.  I was able to get this to crash with the steps you provided.  I'll try to figure out what is going on and provide another update. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] kenballus commented on issue #10116: Segmentation Fault with `attach_server_session_to_client` and Puma backend

Posted by "kenballus (via GitHub)" <gi...@apache.org>.
kenballus commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1658745874

   I have attempted to reproduce this bug with a few other backend servers, but I have not been successful. I am very curious about what exactly Puma is doing with the connection to elicit this response from ATS.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] kenballus commented on issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "kenballus (via GitHub)" <gi...@apache.org>.
kenballus commented on issue #10116:
URL: https://github.com/apache/trafficserver/issues/10116#issuecomment-1722683756

   I can no longer reproduce the issue. Seems like it's fixed. Thanks everyone!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [trafficserver] kenballus closed issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend

Posted by "kenballus (via GitHub)" <gi...@apache.org>.
kenballus closed issue #10116: Segmentation fault with `attach_server_session_to_client` and Puma backend
URL: https://github.com/apache/trafficserver/issues/10116


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org