You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Travis CI <bu...@travis-ci.com> on 2020/11/19 12:33:56 UTC
Broken: apache/httpd#1216 (trunk - 2865f25)
Build Update for apache/httpd
-------------------------------------
Build: #1216
Status: Broken
Duration: 19 mins and 3 secs
Commit: 2865f25 (trunk)
Author: Yann Ylavic
Message: Remove indentation preserved in r1883639 to ease review.
No functional changes.
git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640 13f79535-47bb-0310-9956-ffa450edef68
View the changeset: https://github.com/apache/httpd/compare/7c220fc67527...2865f2584b4d
View the full build log and details: https://travis-ci.com/github/apache/httpd/builds/202714023?utm_medium=notification&utm_source=email
--
You can unsubscribe from build emails from the apache/httpd repository going to https://travis-ci.com/account/preferences/unsubscribe?repository=16806660&utm_medium=notification&utm_source=email.
Or unsubscribe from *all* email updating your settings at https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email.
Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Nov 23, 2020 at 05:16:42PM +0100, Yann Ylavic wrote:
> On Mon, Nov 23, 2020 at 5:06 PM Yann Ylavic <yl...@gmail.com> wrote:
> >
> > Yes apr-1.7.x and apr-util-1.7.x would be nice since we can CTR there
> > quite easily and verify our fixes.
>
> Btw, I merged my APR changes (from today) to 1.7.x already :)
>
Failed in trunk: https://travis-ci.com/github/apache/httpd/builds/203736572
which used APR 1.7.x at r1883753
I also rebased my PR which runs the "ab" TLSv1.2 test onto trunk, and
that actually passed. So, 50% successful? ;)
The pertinent threads from the failed build are, I assume -
Thread 39 (Thread 0x7f988052c800 (LWP 722)):
#0 0x00007f987f794449 in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f987f9c84b7 in run_cleanups (cref=<optimized out>) at memory/unix/apr_pools.c:2672
#2 pool_clear_debug (file_line=<optimized out>, pool=<optimized out>) at memory/unix/apr_pools.c:1859
#3 apr_pool_clear_debug (pool=0x55be404c5a50, file_line=0x7f987dda1398 "event.c:757") at memory/unix/apr_pools.c:1915
#4 0x00007f987f9c877e in pool_destroy_debug (pool=0x55be404c5a50, file_line=<optimized out>) at memory/unix/apr_pools.c:1943
#5 0x00007f987f9c8490 in pool_clear_debug (file_line=<optimized out>, pool=<optimized out>) at memory/unix/apr_pools.c:1856
#6 apr_pool_clear_debug (pool=0x55be408032b0, file_line=0x7f987dda1398 "event.c:757") at memory/unix/apr_pools.c:1915
#7 0x00007f987f9c877e in pool_destroy_debug (pool=0x55be408032b0, file_line=<optimized out>) at memory/unix/apr_pools.c:1943
#8 0x00007f987f9c882c in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1989
#9 0x00007f987dd9c89c in clean_child_exit (code=0) at event.c:757
#10 0x00007f987dd9d546 in child_main (child_num_arg=child_num_arg@entry=3, child_bucket=child_bucket@entry=0) at event.c:2926
#11 0x00007f987dd9d8d4 in make_child (s=0x55be409e2d80, slot=slot@entry=3, bucket=bucket@entry=0) at event.c:2992
#12 0x00007f987dd9e3db in perform_idle_server_maintenance (num_buckets=<optimized out>, child_bucket=<optimized out>) at event.c:3187
#13 server_main_loop (num_buckets=1, remaining_children_to_start=0) at event.c:3316
#14 event_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at event.c:3393
#15 0x000055be3f158b6e in ap_run_mpm (pconf=0x55be40492be0, plog=0x55be404c1620, s=0x55be409e2d80) at mpm_common.c:100
#16 0x000055be3f13bff3 in main (argc=<optimized out>, argv=<optimized out>) at main.c:844
Thread 1 (Thread 0x7f984e2ec700 (LWP 760)):
#0 0x00007f987f3d4e97 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f987f3d6801 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f987f9c8835 in apr_pool_destroy_debug (pool=<optimized out>, file_line=<optimized out>) at memory/unix/apr_pools.c:1987
#3 0x00007f987f9d3546 in apr_thread_exit (thd=thd@entry=0x55be4097b0e0, retval=retval@entry=0) at threadproc/unix/thread.c:206
#4 0x00007f98688e2fe6 in slot_run (thread=0x55be4097b0e0, wctx=0x55be404d90f0) at h2_workers.c:248
#5 0x00007f987f78e6db in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f987f4b788f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Nov 23, 2020 at 5:06 PM Yann Ylavic <yl...@gmail.com> wrote:
>
> Yes apr-1.7.x and apr-util-1.7.x would be nice since we can CTR there
> quite easily and verify our fixes.
Btw, I merged my APR changes (from today) to 1.7.x already :)
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Nov 23, 2020 at 2:33 PM Joe Orton <jo...@redhat.com> wrote:
>
> On Mon, Nov 23, 2020 at 01:24:39PM +0100, Yann Ylavic wrote:
> > --- srclib/apr-trunk/memory/unix/apr_pools.c (revision 1883742)
> > +++ srclib/apr-trunk/memory/unix/apr_pools.c (working copy)
> > @@ -1951,10 +1951,8 @@ APR_DECLARE(void) apr_pool_clear_debug(apr_pool_t
> >
> > static void pool_destroy_debug(apr_pool_t *pool, const char *file_line)
> > {
> > - apr_pool_check_lifetime(pool);
> > + apr_pool_clear_debug(pool, file_line);
> >
> > - pool_clear_debug(pool, file_line);
> > -
> > #if (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE)
> > apr_pool_log_event(pool, "DESTROY", file_line, 1);
> > #endif /* (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE) */):
> > --
> >
> > No more use-after-free, while without this patch there are many ones
> > (like in the attached ASAN report).
> > Note: pool_clear_debug is not mutex protected, while apr_pool_clear_debug is.
>
> Nice work!
Thanks, most credits to Rüdiger who suggested it!
>
> > Could we run APR_POOL_DEBUG test --with-inclued-apr (only) on travis,
> > or fetching the sources causes issues still?
>
> Do you mean a different APR/APR-util tag/branch than 1.7.0/1.6.1? The
> only restriction is that we can't test LDAP/pool-debug (usefully) if
> building against APR trunk. We can switch to building against 1.7.x
> branches, it works but adds a few minutes to the build time because we
> can't cache the installs in that specific case.
Yes apr-1.7.x and apr-util-1.7.x would be nice since we can CTR there
quite easily and verify our fixes.
>
> BTW do you have an idiots guide to building/testing httpd with ASAN? We
> could add that to travis too.
Attached is my config.nice, I'm using ASAN from gcc-9 because gcc-10's
is buggy for me (it somehow strips the link with libcrypt.so, without
failing the build, and then crashes in apr_password_validate() when
crypt_r() is called at a NULL address..).
Then I run the perl test framework like this:
$ (ulimit -c unlimited; export
LSAN_OPTIONS=halt_on_error=1:disable_coredump=0:unmap_shadow_on_exit=1;
./t/TEST)
Although the above LSAN_OPTIONS options are supposed to coredump
whenever libasan detects something, it does not work for me (possibly
has something to do with ASAN and httpd competing SEGV handlers..).
The CFLAGS's -fsanitize-recover=address in addition to
-fsanitize=address was an attempt to get coredumps too, not it's
needed for the ci.
Anyway, by default ASAN puts its reports on stderr, so I filter afterward with:
$ grep -v "^\[" t/logs/error_log |less
I think adding ":log_path=/path/to/asan/report.txt" to LSAN_OPTIONS
could make it log wherever we want too (never tried though..).
I wish I could make ASAN dump cores to be able to see all the threads
when an issue occurs (and have a wider picture), because ASAN reports
show only the two threads concerned by the faulty address. No luck
with that on my machine so far, but it can very well be a local issue
only.
The two threads were enough for me to figure out so far..
Hth;
Yann.
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Nov 23, 2020 at 01:24:39PM +0100, Yann Ylavic wrote:
> --- srclib/apr-trunk/memory/unix/apr_pools.c (revision 1883742)
> +++ srclib/apr-trunk/memory/unix/apr_pools.c (working copy)
> @@ -1951,10 +1951,8 @@ APR_DECLARE(void) apr_pool_clear_debug(apr_pool_t
>
> static void pool_destroy_debug(apr_pool_t *pool, const char *file_line)
> {
> - apr_pool_check_lifetime(pool);
> + apr_pool_clear_debug(pool, file_line);
>
> - pool_clear_debug(pool, file_line);
> -
> #if (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE)
> apr_pool_log_event(pool, "DESTROY", file_line, 1);
> #endif /* (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE) */):
> --
>
> No more use-after-free, while without this patch there are many ones
> (like in the attached ASAN report).
> Note: pool_clear_debug is not mutex protected, while apr_pool_clear_debug is.
Nice work!
> Could we run APR_POOL_DEBUG test --with-inclued-apr (only) on travis,
> or fetching the sources causes issues still?
Do you mean a different APR/APR-util tag/branch than 1.7.0/1.6.1? The
only restriction is that we can't test LDAP/pool-debug (usefully) if
building against APR trunk. We can switch to building against 1.7.x
branches, it works but adds a few minutes to the build time because we
can't cache the installs in that specific case.
BTW do you have an idiots guide to building/testing httpd with ASAN? We
could add that to travis too.
Regards, Joe
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Nov 23, 2020 at 11:12 AM Yann Ylavic <yl...@gmail.com> wrote:
>
> I think that apr-1.7.0 and apr-util-1.6.1 are missing r1481186 and
> r1822315 respectively. Both fixed quite some APR_POOL_DEBUG
> correctness according to my late ASAN testing.
> They were backported already, but there was no apr[-util] releases since then.
Nope, didn't help.
>
> I'll try running ab with and without them, and see how it goes..
> Also, Rüdiger (in some thread) proposed to lock pool_destroy_debug()
> like it's done for pool_clear_debug() already, that looks interesting
> too.
That helped! Stressing with ab and using this patch:
Index: srclib/apr-trunk/memory/unix/apr_pools.c
===================================================================
--- srclib/apr-trunk/memory/unix/apr_pools.c (revision 1883742)
+++ srclib/apr-trunk/memory/unix/apr_pools.c (working copy)
@@ -1951,10 +1951,8 @@ APR_DECLARE(void) apr_pool_clear_debug(apr_pool_t
static void pool_destroy_debug(apr_pool_t *pool, const char *file_line)
{
- apr_pool_check_lifetime(pool);
+ apr_pool_clear_debug(pool, file_line);
- pool_clear_debug(pool, file_line);
-
#if (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE)
apr_pool_log_event(pool, "DESTROY", file_line, 1);
#endif /* (APR_POOL_DEBUG & APR_POOL_DEBUG_VERBOSE) */):
--
No more use-after-free, while without this patch there are many ones
(like in the attached ASAN report).
Note: pool_clear_debug is not mutex protected, while apr_pool_clear_debug is.
Could we run APR_POOL_DEBUG test --with-inclued-apr (only) on travis,
or fetching the sources causes issues still?
Not sure distros pick up APR_POOL_DEBUG changes in their libapr..
Or we might simply accept ci sporadic failures with APR_POOL_DEBUG until then.
Regards;
Yann.
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Yann Ylavic <yl...@gmail.com>.
On Mon, Nov 23, 2020 at 10:17 AM Joe Orton <jo...@redhat.com> wrote:
>
> On Fri, Nov 20, 2020 at 06:08:18PM +0100, Yann Ylavic wrote:
> > I tried to reproduce this failure locally (debian buster) with multiple
> > tries but couldn't.
> > I think I'm using the same environment (apr-1.7.0, apr-util-1.6.1,
> > APR_POOL_DEBUG, docker.io/redis:latest), and in the same way running:
> > $ (export SSL_SESSCACHE=redis:localhost:6379; ./t/TEST -sslproto TLSv1.2
> > -defines TEST_SSL_SESSCACHE t/ssl)
>
> There was a thread on this discussing whether apr_pool_find() was really safe...
Yeah, I saw it after writing this message.
>
> I think that's the right setup, but it did not *reliably* fail just by
> running t/TEST alone. IIRC it did reliably fail by starting the server
> as in your command above, and then running "ab" against the server:
>
> ab -qd -n 4000 -c 20 -f TLS1.2 https://localhost:8532/
>
> and https://github.com/apache/httpd/pull/140 showed this failure was
> specific to pool-debug. I could reliably reproduce it locally as well.
I will try this.
I think that apr-1.7.0 and apr-util-1.6.1 are missing r1481186 and
r1822315 respectively. Both fixed quite some APR_POOL_DEBUG
correctness according to my late ASAN testing.
They were backported already, but there was no apr[-util] releases since then.
I'll try running ab with and without them, and see how it goes..
Also, Rüdiger (in some thread) proposed to lock pool_destroy_debug()
like it's done for pool_clear_debug() already, that looks interesting
too.
>
> > Since session caching is not a failure point in mod_ssl, I also verified
> > with redis up or down (and checked traffic with tcpdump), no luck either.
> > Does it happen only with socache_redis?
> > These seem to be the two threads racing:
>
> I don't think it is specific to redis, there was a similar failure in:
> https://travis-ci.org/github/apache/httpd/jobs/723103332 which used
> memcache for example.
>
> That said it's interesting and maybe significant if it doesn't fail with
> shmcb?
Let's see, thanks Joe.
Regards;
Yann.
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Nov 20, 2020 at 06:08:18PM +0100, Yann Ylavic wrote:
> I tried to reproduce this failure locally (debian buster) with multiple
> tries but couldn't.
> I think I'm using the same environment (apr-1.7.0, apr-util-1.6.1,
> APR_POOL_DEBUG, docker.io/redis:latest), and in the same way running:
> $ (export SSL_SESSCACHE=redis:localhost:6379; ./t/TEST -sslproto TLSv1.2
> -defines TEST_SSL_SESSCACHE t/ssl)
There was a thread on this discussing whether apr_pool_find() was really safe...
I think that's the right setup, but it did not *reliably* fail just by
running t/TEST alone. IIRC it did reliably fail by starting the server
as in your command above, and then running "ab" against the server:
ab -qd -n 4000 -c 20 -f TLS1.2 https://localhost:8532/
and https://github.com/apache/httpd/pull/140 showed this failure was
specific to pool-debug. I could reliably reproduce it locally as well.
> Since session caching is not a failure point in mod_ssl, I also verified
> with redis up or down (and checked traffic with tcpdump), no luck either.
> Does it happen only with socache_redis?
> These seem to be the two threads racing:
I don't think it is specific to redis, there was a similar failure in:
https://travis-ci.org/github/apache/httpd/jobs/723103332 which used
memcache for example.
That said it's interesting and maybe significant if it doesn't fail with
shmcb?
Regards, Joe
>
> Thread 41 (Thread 0x7f59de0c7700 (LWP 32462)):
> #0 0x00007f5a19215b39 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f5a1921ac27 in free () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f5a197b5268 in pool_clear_debug (pool=pool@entry=0x55a892d7d9b0,
> file_line=<optimized out>) at memory/unix/apr_pools.c:1853
> #3 0x00007f5a197b4fee in pool_destroy_debug (pool=0x55a892d7d9b0,
> file_line=<optimized out>) at memory/unix/apr_pools.c:1915
> #4 0x00007f5a197b585c in apr_pool_destroy_debug (pool=<optimized out>,
> file_line=<optimized out>) at memory/unix/apr_pools.c:1957
> #5 0x000055a8929613c5 in eor_bucket_destroy (data=0x55a8931b87d8) at
> eor_bucket.c:108
> #6 0x000055a89296230b in send_brigade_nonblocking (c=0x55a892d84a20,
> ctx=0x55a8930b5a30, bb=0x55a892f72ac0, s=0x55a892d7d430) at
> core_filters.c:570
> #7 ap_core_output_filter (f=0x55a892f3d3c0, bb=0x55a892f72ac0) at
> core_filters.c:407
> #8 0x00007f5a03e76098 in bio_filter_out_pass
> (outctx=outctx@entry=0x55a892f747a0)
> at ssl_engine_io.c:139
> #9 0x00007f5a03e799e4 in ssl_io_filter_output (f=0x55a892f746e0,
> bb=0x55a892f711a0) at ssl_engine_io.c:1988
> #10 0x00007f5a03e767a8 in ssl_io_filter_coalesce (f=0x55a892f72c40,
> bb=0x55a892f711a0) at ssl_engine_io.c:1922
> #11 0x000055a89292dc6e in ap_process_request_after_handler (r=<optimized
> out>) at http_request.c:375
> #12 0x000055a89292b731 in ap_process_http_async_connection
> (c=0x55a892d84a20) at http_core.c:158
> #13 ap_process_http_connection (c=0x55a892d84a20) at http_core.c:252
> #14 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d84a20)
> at connection.c:42
> #15 0x00007f5a17b8c65d in process_socket (thd=0x55a89313fcc0, p=<optimized
> out>, sock=<optimized out>, cs=0x55a892d0d650, my_child_num=<optimized
> out>, my_thread_num=<optimized out>) at event.c:1094
> #16 0x00007f5a17b8cf8f in worker_thread (thd=0x55a89313fcc0,
> dummy=<optimized out>) at event.c:2383
> #17 0x00007f5a1957b6db in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #18 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> Thread 66 (Thread 0x7f59df0c9700 (LWP 32460)):
> #0 0x00007f5a191c2187 in kill () from /lib/x86_64-linux-gnu/libc.so.6
> #1 <signal handler called>
> #2 0x00007f5a191c1e97 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x00007f5a191c3801 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #4 0x00007f5a197adf62 in apr_table_mergen (t=0x55a89306dfb0,
> key=key@entry=0x55a8929870aa
> "Connection", val=val@entry=0x55a89297ff86 "close") at
> tables/apr_tables.c:750
> #5 0x000055a89292bfff in ap_set_keepalive (r=r@entry=0x55a892fd2540) at
> http_protocol.c:309
> #6 0x000055a892931435 in ap_http_header_filter (f=0x55a892f39c30,
> b=0x55a892f54ea0) at http_filters.c:1372
> #7 0x000055a89294f071 in ap_content_length_filter (f=0x55a892f38140,
> b=0x55a892f54ea0) at protocol.c:2040
> #8 0x000055a8929332a7 in ap_byterange_filter (f=0x55a892d22370,
> bb=0x55a892f54ea0) at byterange_filter.c:463
> #9 0x00007f5a04ab8e14 in session_output_filter (f=0x55a892f3d8b0,
> in=0x55a892f54ea0) at mod_session.c:501
> #10 0x00007f5a05efdc10 in ap_proxy_http_process_response
> (req=req@entry=0x55a892f46bf0)
> at mod_proxy_http.c:1997
> #11 0x00007f5a05f0018f in proxy_http_handler (r=<optimized out>,
> worker=<optimized out>, conf=0x55a892e614f0, url=<optimized out>,
> proxyname=0x0, proxyport=<optimized out>) at mod_proxy_http.c:2326
> #12 0x00007f5a0651f7ec in proxy_run_scheme_handler (r=r@entry=0x55a892fd2540,
> worker=0x55a892e61780, conf=conf@entry=0x55a892e614f0, url=0x55a89325ea96 "
> https://localhost:8532/", proxyhost=proxyhost@entry=0x0,
> proxyport=proxyport@entry=0) at mod_proxy.c:3519
> #13 0x00007f5a0652189e in proxy_handler (r=0x55a892fd2540) at
> mod_proxy.c:1525
> #14 0x000055a892935b00 in ap_run_handler (r=r@entry=0x55a892fd2540) at
> config.c:170
> #15 0x000055a89293609d in ap_invoke_handler (r=r@entry=0x55a892fd2540) at
> config.c:444
> #16 0x000055a89292ec9b in ap_process_async_request (r=0x55a892fd2540) at
> http_request.c:463
> #17 0x000055a89292b731 in ap_process_http_async_connection
> (c=0x55a892d13490) at http_core.c:158
> #18 ap_process_http_connection (c=0x55a892d13490) at http_core.c:252
> #19 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d13490)
> at connection.c:42
> #20 0x00007f5a17b8c65d in process_socket (thd=0x55a8931410b0, p=<optimized
> out>, sock=<optimized out>, cs=0x55a892cfe760, my_child_num=<optimized
> out>, my_thread_num=<optimized out>) at event.c:1094
> #21 0x00007f5a17b8cf8f in worker_thread (thd=0x55a8931410b0,
> dummy=<optimized out>) at event.c:2383
> #22 0x00007f5a1957b6db in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #23 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6
>
> But we don't in Thread #41 whether the request is the same as in Thread #66
> (0x55a892fd2540)..
>
> Any idea on how to reproduce it? I'm out of ideas currently :/
>
> Regards;
> Yann.
>
> On Thu, Nov 19, 2020 at 1:34 PM Travis CI <bu...@travis-ci.com> wrote:
> >
> > apache / httpd
> >
> > trunk
> >
> > Build #1216 was broken
> > 19 mins and 3 secs
> > Yann Ylavic
> > 2865f25 CHANGESET →
> >
> > Remove indentation preserved in r1883639 to ease review.
> >
> > No functional changes.
> >
> >
> > git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640
> 13f79535-47bb-0310-9956-ffa450edef68
> >
> > Want to know about upcoming build environment updates?
> >
> > Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you!
> >
> > SIGN UP HERE
> > Documentation about Travis CI
> > Have any questions? We're here to help.
> > Unsubscribe from build emails from the apache/httpd repository.
> > To unsubscribe from all build emails, please update your settings.
> >
>
>
> On Thu, Nov 19, 2020 at 1:34 PM Travis CI <bu...@travis-ci.com> wrote:
>
> > apache
> >
> > /
> >
> > httpd
> >
> > <https://travis-ci.com/github/apache/httpd?utm_medium=notification&utm_source=email>
> >
> > [image: branch icon]trunk <https://github.com/apache/httpd/tree/trunk>
> > [image: build has failed]
> > Build #1216 was broken
> > <https://travis-ci.com/github/apache/httpd/builds/202714023?utm_medium=notification&utm_source=email>
> > [image: arrow to build time]
> > [image: clock icon]19 mins and 3 secs
> >
> > [image: Yann Ylavic avatar]Yann Ylavic
> > 2865f25 CHANGESET →
> > <https://github.com/apache/httpd/compare/7c220fc67527...2865f2584b4d>
> >
> > Remove indentation preserved in r1883639 to ease review.
> >
> > No functional changes.
> >
> >
> > git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640
> > 13f79535-47bb-0310-9956-ffa450edef68
> >
> > Want to know about upcoming build environment updates?
> >
> > Would you like to stay up-to-date with the upcoming Travis CI build
> > environment updates? We set up a mailing list for you!
> > SIGN UP HERE <http://eepurl.com/9OCsP>
> >
> > [image: book icon]
> >
> > Documentation <https://docs.travis-ci.com/> about Travis CI
> > Have any questions? We're here to help. <su...@travis-ci.com>
> > Unsubscribe
> > <https://travis-ci.com/account/preferences/unsubscribe?repository=16806660&utm_medium=notification&utm_source=email>
> > from build emails from the apache/httpd repository.
> > To unsubscribe from *all* build emails, please update your settings
> > <https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email>.
> >
> > [image: black and white travis ci logo] <https://travis-ci.com>
> >
> > Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
> > Jacops | Contact: contact@travis-ci.com | Amtsgericht Charlottenburg,
> > Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß §27 a Umsatzsteuergesetz:
> > DE282002648
> >
Re: Broken: apache/httpd#1216 (trunk - 2865f25)
Posted by Yann Ylavic <yl...@gmail.com>.
I tried to reproduce this failure locally (debian buster) with multiple
tries but couldn't.
I think I'm using the same environment (apr-1.7.0, apr-util-1.6.1,
APR_POOL_DEBUG, docker.io/redis:latest), and in the same way running:
$ (export SSL_SESSCACHE=redis:localhost:6379; ./t/TEST -sslproto TLSv1.2
-defines TEST_SSL_SESSCACHE t/ssl)
Since session caching is not a failure point in mod_ssl, I also verified
with redis up or down (and checked traffic with tcpdump), no luck either.
Does it happen only with socache_redis?
These seem to be the two threads racing:
Thread 41 (Thread 0x7f59de0c7700 (LWP 32462)):
#0 0x00007f5a19215b39 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f5a1921ac27 in free () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f5a197b5268 in pool_clear_debug (pool=pool@entry=0x55a892d7d9b0,
file_line=<optimized out>) at memory/unix/apr_pools.c:1853
#3 0x00007f5a197b4fee in pool_destroy_debug (pool=0x55a892d7d9b0,
file_line=<optimized out>) at memory/unix/apr_pools.c:1915
#4 0x00007f5a197b585c in apr_pool_destroy_debug (pool=<optimized out>,
file_line=<optimized out>) at memory/unix/apr_pools.c:1957
#5 0x000055a8929613c5 in eor_bucket_destroy (data=0x55a8931b87d8) at
eor_bucket.c:108
#6 0x000055a89296230b in send_brigade_nonblocking (c=0x55a892d84a20,
ctx=0x55a8930b5a30, bb=0x55a892f72ac0, s=0x55a892d7d430) at
core_filters.c:570
#7 ap_core_output_filter (f=0x55a892f3d3c0, bb=0x55a892f72ac0) at
core_filters.c:407
#8 0x00007f5a03e76098 in bio_filter_out_pass
(outctx=outctx@entry=0x55a892f747a0)
at ssl_engine_io.c:139
#9 0x00007f5a03e799e4 in ssl_io_filter_output (f=0x55a892f746e0,
bb=0x55a892f711a0) at ssl_engine_io.c:1988
#10 0x00007f5a03e767a8 in ssl_io_filter_coalesce (f=0x55a892f72c40,
bb=0x55a892f711a0) at ssl_engine_io.c:1922
#11 0x000055a89292dc6e in ap_process_request_after_handler (r=<optimized
out>) at http_request.c:375
#12 0x000055a89292b731 in ap_process_http_async_connection
(c=0x55a892d84a20) at http_core.c:158
#13 ap_process_http_connection (c=0x55a892d84a20) at http_core.c:252
#14 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d84a20)
at connection.c:42
#15 0x00007f5a17b8c65d in process_socket (thd=0x55a89313fcc0, p=<optimized
out>, sock=<optimized out>, cs=0x55a892d0d650, my_child_num=<optimized
out>, my_thread_num=<optimized out>) at event.c:1094
#16 0x00007f5a17b8cf8f in worker_thread (thd=0x55a89313fcc0,
dummy=<optimized out>) at event.c:2383
#17 0x00007f5a1957b6db in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#18 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 66 (Thread 0x7f59df0c9700 (LWP 32460)):
#0 0x00007f5a191c2187 in kill () from /lib/x86_64-linux-gnu/libc.so.6
#1 <signal handler called>
#2 0x00007f5a191c1e97 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f5a191c3801 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f5a197adf62 in apr_table_mergen (t=0x55a89306dfb0,
key=key@entry=0x55a8929870aa
"Connection", val=val@entry=0x55a89297ff86 "close") at
tables/apr_tables.c:750
#5 0x000055a89292bfff in ap_set_keepalive (r=r@entry=0x55a892fd2540) at
http_protocol.c:309
#6 0x000055a892931435 in ap_http_header_filter (f=0x55a892f39c30,
b=0x55a892f54ea0) at http_filters.c:1372
#7 0x000055a89294f071 in ap_content_length_filter (f=0x55a892f38140,
b=0x55a892f54ea0) at protocol.c:2040
#8 0x000055a8929332a7 in ap_byterange_filter (f=0x55a892d22370,
bb=0x55a892f54ea0) at byterange_filter.c:463
#9 0x00007f5a04ab8e14 in session_output_filter (f=0x55a892f3d8b0,
in=0x55a892f54ea0) at mod_session.c:501
#10 0x00007f5a05efdc10 in ap_proxy_http_process_response
(req=req@entry=0x55a892f46bf0)
at mod_proxy_http.c:1997
#11 0x00007f5a05f0018f in proxy_http_handler (r=<optimized out>,
worker=<optimized out>, conf=0x55a892e614f0, url=<optimized out>,
proxyname=0x0, proxyport=<optimized out>) at mod_proxy_http.c:2326
#12 0x00007f5a0651f7ec in proxy_run_scheme_handler (r=r@entry=0x55a892fd2540,
worker=0x55a892e61780, conf=conf@entry=0x55a892e614f0, url=0x55a89325ea96 "
https://localhost:8532/", proxyhost=proxyhost@entry=0x0,
proxyport=proxyport@entry=0) at mod_proxy.c:3519
#13 0x00007f5a0652189e in proxy_handler (r=0x55a892fd2540) at
mod_proxy.c:1525
#14 0x000055a892935b00 in ap_run_handler (r=r@entry=0x55a892fd2540) at
config.c:170
#15 0x000055a89293609d in ap_invoke_handler (r=r@entry=0x55a892fd2540) at
config.c:444
#16 0x000055a89292ec9b in ap_process_async_request (r=0x55a892fd2540) at
http_request.c:463
#17 0x000055a89292b731 in ap_process_http_async_connection
(c=0x55a892d13490) at http_core.c:158
#18 ap_process_http_connection (c=0x55a892d13490) at http_core.c:252
#19 0x000055a892944d30 in ap_run_process_connection (c=c@entry=0x55a892d13490)
at connection.c:42
#20 0x00007f5a17b8c65d in process_socket (thd=0x55a8931410b0, p=<optimized
out>, sock=<optimized out>, cs=0x55a892cfe760, my_child_num=<optimized
out>, my_thread_num=<optimized out>) at event.c:1094
#21 0x00007f5a17b8cf8f in worker_thread (thd=0x55a8931410b0,
dummy=<optimized out>) at event.c:2383
#22 0x00007f5a1957b6db in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007f5a192a488f in clone () from /lib/x86_64-linux-gnu/libc.so.6
But we don't in Thread #41 whether the request is the same as in Thread #66
(0x55a892fd2540)..
Any idea on how to reproduce it? I'm out of ideas currently :/
Regards;
Yann.
On Thu, Nov 19, 2020 at 1:34 PM Travis CI <bu...@travis-ci.com> wrote:
>
> apache / httpd
>
> trunk
>
> Build #1216 was broken
> 19 mins and 3 secs
> Yann Ylavic
> 2865f25 CHANGESET →
>
> Remove indentation preserved in r1883639 to ease review.
>
> No functional changes.
>
>
> git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640
13f79535-47bb-0310-9956-ffa450edef68
>
> Want to know about upcoming build environment updates?
>
> Would you like to stay up-to-date with the upcoming Travis CI build
environment updates? We set up a mailing list for you!
>
> SIGN UP HERE
> Documentation about Travis CI
> Have any questions? We're here to help.
> Unsubscribe from build emails from the apache/httpd repository.
> To unsubscribe from all build emails, please update your settings.
>
On Thu, Nov 19, 2020 at 1:34 PM Travis CI <bu...@travis-ci.com> wrote:
> apache
>
> /
>
> httpd
>
> <https://travis-ci.com/github/apache/httpd?utm_medium=notification&utm_source=email>
>
> [image: branch icon]trunk <https://github.com/apache/httpd/tree/trunk>
> [image: build has failed]
> Build #1216 was broken
> <https://travis-ci.com/github/apache/httpd/builds/202714023?utm_medium=notification&utm_source=email>
> [image: arrow to build time]
> [image: clock icon]19 mins and 3 secs
>
> [image: Yann Ylavic avatar]Yann Ylavic
> 2865f25 CHANGESET →
> <https://github.com/apache/httpd/compare/7c220fc67527...2865f2584b4d>
>
> Remove indentation preserved in r1883639 to ease review.
>
> No functional changes.
>
>
> git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1883640
> 13f79535-47bb-0310-9956-ffa450edef68
>
> Want to know about upcoming build environment updates?
>
> Would you like to stay up-to-date with the upcoming Travis CI build
> environment updates? We set up a mailing list for you!
> SIGN UP HERE <http://eepurl.com/9OCsP>
>
> [image: book icon]
>
> Documentation <https://docs.travis-ci.com/> about Travis CI
> Have any questions? We're here to help. <su...@travis-ci.com>
> Unsubscribe
> <https://travis-ci.com/account/preferences/unsubscribe?repository=16806660&utm_medium=notification&utm_source=email>
> from build emails from the apache/httpd repository.
> To unsubscribe from *all* build emails, please update your settings
> <https://travis-ci.com/account/preferences/unsubscribe?utm_medium=notification&utm_source=email>.
>
> [image: black and white travis ci logo] <https://travis-ci.com>
>
> Travis CI GmbH, Rigaer Str. 8, 10427 Berlin, Germany | GF/CEO: Randy
> Jacops | Contact: contact@travis-ci.com | Amtsgericht Charlottenburg,
> Berlin, HRB 140133 B | Umsatzsteuer-ID gemäß §27 a Umsatzsteuergesetz:
> DE282002648
>