You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pagespeed.apache.org by GitBox <gi...@apache.org> on 2020/09/21 22:28:52 UTC
[GitHub] [incubator-pagespeed-mod] ajconejo opened a new issue #2041: mod_pagespeed chrashes from time to time
ajconejo opened a new issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041
Hi
Cheking apache error log I found that pagesepeed crashes from time to time.
Any idea what would be the cause?
Crash log is below as well as some other errors found.
Ubunu 18.04
pagespeed 1.13.35.2-0
Full pagespeed config:
Version: 14: on
Filters
ah Add Head
ij Canonicalize Javascript library URLs
cc Combine Css
jc Combine Javascript
gp Convert Gif to Png
jp Convert Jpeg to Progressive
jw Convert Jpeg To Webp
mc Convert Meta Tags
pj Convert Png to Jpeg
ws When converting images to WebP, prefer lossless conversions
dd Dedup Inlined Images
di Delay Images
ec Cache Extend Css
ei Cache Extend Images
es Cache Extend Scripts
fc Fallback Rewrite Css
if Flatten CSS Imports
hpsr Hint Preload of Subresources
hw Flushes html
ci Inline Css
gf Inline Google Font CSS
ii Inline Images
il Inline @import to Link
ji Inline Javascript
io In-place optimize for browser
idp Insert DNS Prefetch
id Insert Image Dimensions
js Jpeg Subsampling
co Outline Css
jo Outline Javascript
rj Recompress Jpeg
rp Recompress Png
rw Recompress Webp
rc Remove Comments
ri Resize Images
rm Resize Mobile Images
cf Rewrite Css
jm Rewrite External Javascript
jj Rewrite Inline Javascript
cu Rewrite Style Attributes With Url
cp Strip Image Color Profiles
md Strip Image Meta Data
Options
CacheFlushPollIntervalSec (acfpi) 1296000
CssOutlineMinBytes (co) 3000
EnableCachePurge (euci) True
EnableRewriting (e) 1
FetchHttps (fhs) enable
FileCacheCleanIntervalMs (afcci) 1296000000
FileCacheInodeLimit (afcl) 5000000
FileCachePath (afcp) /var/cache/mod_pagespeed/
FileCacheSizeKb (afc) 512000
GoogleFontCssInlineMaxBytes (gfci) 100000
HttpCacheCompressionLevel (hccl) 9
ImageMaxRewritesAtOnce (im) 10
JsOutlineMinBytes (jo) 3000
LogDir (ald) /var/log/pagespeed
MaxCacheableContentLength (rcl) 20000000
MaxCombinedCssBytes (xcc) -1
MaxCombinedJsBytes (xcj) -1
MemcachedServers (ams) localhost:11212
PurgeMethod (pm) PURGE
RateLimitBackgroundFetches (rlbf) False
RewriteRandomDropPercentage (rrdp) -1
ShmMetadataCacheCheckpointIntervalSec (smci) 300
SslCertDirectory (assld) /etc/ssl/certs
StatisticsLogging (asle) True
Domain Lawyer
http*://*.googletagmanager.com/ Auth
http*://*.ssl-images-amazon.com/ Auth
http*://45.79.198.124/ Auth
http*://biblioteca-admin.celacp.org/ Auth
http*://biblioteca.celacp.org/ Auth
Invalidation Timestamp: (none)
Apache error log extract:
09:27:36.842153 2020] [pagespeed:alert] [pid 13280] [mod_pagespeed 1.13.35.2-0 @13280] [0921/092736:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f08f802970e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f08f8021060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f08f802b8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f08f81c4513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f08f81c465b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f08f8166ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f08f8166f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f08f81c404b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f08f82e9677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f08f82f0d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e) [0x7f08f80
1902e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f08f8019129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f08f8022b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f08fc9270f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f08fc9271ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f08f91979bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x561368e3eaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f08f8d8a831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f08f8d8ab34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7f08f8d8b57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x561368e182be]\n/usr/sbin/apache2(main+0x9fb) [0x561368e10b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f08fc905b97]\n/usr/sbin/apache2(_start+0x2a) [0x561368e10c5a]
[Mon Sep 21 15:03:40.178754 2020] [pagespeed:alert] [pid 335] [mod_pagespeed 1.13.35.2-0 @335] [0921/150340:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f08f802970e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f08f8021060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f08f802b8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f08f81c4513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f08f81c465b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f08f8166ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f08f8166f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f08f81c404b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f08f82e9677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f08f82f0d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e) [0
x7f08f801902e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f08f8019129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f08f8022b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f08fc9270f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f08fc9271ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f08f91979bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x561368e3eaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f08f8d8a831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f08f8d8ab34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7f08f8d8b57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x561368e182be]\n/usr/sbin/apache2(main+0x9fb) [0x561368e10b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f08fc905b97]\n/usr/sbin/apache2(_start+0x2a) [0x561368e10c5a]
[Mon Sep 21 07:01:32.578504 2020] [pagespeed:warn] [pid 7381] [mod_pagespeed 1.13.35.2-0 @7381] [0921/070132:WARNING:serf_url_async_fetcher.cc(1222)] Aborting f
etch of http://45.79.198.124/intranet-tmpl/lib/bootstrap/bootstrap.min_19.1109000.css
[Mon Sep 21 07:01:32.584071 2020] [pagespeed:warn] [pid 7381] [mod_pagespeed 1.13.35.2-0 @7381] [0921/070132:WARNING:serf_url_async_fetcher.cc(1222)] Aborting f
etch of http://45.79.198.124/intranet-tmpl/lib/font-awesome/css/font-awesome.min_19.1109000.css
[Mon Sep 21 07:01:32.586153 2020] [pagespeed:warn] [pid 7381] [mod_pagespeed 1.13.35.2-0 @7381] [0921/070132:WARNING:serf_url_async_fetcher.cc(1222)] Aborting f
etch of http://45.79.198.124/intranet-tmpl/lib/datatables/datatables.min_19.1109000.css
[Mon Sep 21 07:07:39.187163 2020] [pagespeed:warn] [pid 7664] [mod_pagespeed 1.13.35.2-0 @7664] [0921/070739:WARNING:queued_worker_pool.cc(435)] Canceling 1 fun
ctions on sequence Shutdown
[Mon Sep 21 07:11:11.492962 2020] [pagespeed:warn] [pid 7780] [mod_pagespeed 1.13.35.2-0 @7780] [0921/071111:WARNING:queued_worker_pool.cc(435)] Canceling 2 fun
ctions on sequence Shutdown
[Mon Sep 21 07:12:36.957015 2020] [pagespeed:warn] [pid 7839] [mod_pagespeed 1.13.35.2-0 @7839] [0921/071236:WARNING:queued_worker_pool.cc(435)] Canceling 1 fun
ctions on sequence Shutdown
[Mon Sep 21 12:44:08.317046 2020] [pagespeed:error] [pid 25142] [mod_pagespeed 1.13.35.2-0 @25142] AprMemCache::Put error: Internal error (specific information
not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js, value-size 901755
[Mon Sep 21 12:44:09.604895 2020] [pagespeed:error] [pid 25142] [mod_pagespeed 1.13.35.2-0 @25142] AprMemCache::Put error: Internal error (specific information
not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js, value-size 901755
[Mon Sep 21 12:44:09.880243 2020] [pagespeed:error] [pid 25142] [mod_pagespeed 1.13.35.2-0 @25142] AprMemCache::Put error: Internal error (specific information
not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js.pagespeed.jm.gahYYWZ2g
b.js, value-size 899910
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704593740
> Hi -- thanks for the stack-traces. I do not remember seeing this problem before, but would you be willing to try 2 things:
>
> 1. upgrade to mod_pagespeed 1.14 to see if this resolves the problem
> 2. switch from memcached to redis
>
> I can't promise either of these fixes the issue, but it would be good at least to confirm whether (a) this was fixed already in 1.14 and (b) whether this is specific to memcached or common to the redis/memcache implementations. If it is memcached-specific then you might be happy with Redis as an alternative anyway, and in either case we'll know more once you switch.
>
> In case that does not work, I've looked at the code, and this was a feature added to coalesce multiple asynchronous reads of the same key. I think in retrospect this should've been made an option in case something like this went awry. It might be possible to disable or workaround this code, which was added 4 years ago in this commit: [42098f8](https://github.com/apache/incubator-pagespeed-mod/commit/42098f8ed9398ef1ce37a337c2aee2bdd61ee946) .
>
> Ideally if anyone has time and wants a C++ challenge, trying to repro and fix this would be a great contribution!
Hi JMarantz
I´ll try the two options you propose. I guess 1.14 is the devel version?
As per documentation, seems redis support is still in beta. Is that right?
Regards
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704086305
Have you set `ModPagespeedMemcachedServers localhost:11211` in pagespeed config? Is the memcached instance running?
This error `Could not find specified socket in poll list. (70015)` is related to be unable to connect to the memcached instance.
Are tou able to do:
`telnet localhost:11211` and run the command `stats` (for example)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-699524768
Hi Lofesa
Thanks for your time.
I´ve disabled memcached from pagespeed and are monitorig the issue. As we are on weekend no one will work with the system so I would need to wait until Monday/Tuesday to see if there is a difference.
I´ve however, added the idle_timeout variable to memcached config file since I have memcached v1.5.6.
About the hundreds of connections, I my guess is that either memcached, pagespeed or apache are not handling connections correctly or is a config error somewhere.
As I´m running two memcahed instances and pagespeed has its own instance. The rest of memcached clients use the other instance.
This is a very small system with no more than 3 or 4 users at a time and I doudbt that they can generate that much connections.
Will update thread sometime next week
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz edited a comment on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
jmarantz edited a comment on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704596443
1.14 is a beta release. I actually don't think it's *that* likely this problem was fixed in it, but it's worth a shot.
Redis support may be in "beta" but it may not suffer from the same underlying issue that triggers the coalescing failure in memcached. The problem is -- and Lofesa pointed this out -- that the only reason we are getting into the place where mod_pagespeed crashes is that first, the connection to memcached failed. I do not know why that failed, but the underlying network connection to redis is actually a lot simpler and less likely to fail in the same way. So I think Redis is worth a shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-697107059
Hi Otto
Thanks for your time and support
I´ve enabled RateLimitBGFetches to see what happens.
About LoadFromFile, I don´t used because, for some reason if enabled, I start to have a high rate of Caches misses (over 50%) and ressources not rewritten because of cache headers (over 75%).
About the crashes, I don´t thnk is related to shutdown since apache was not restarted at the times shown on the logs. Or do you mean another type of shitdown? Any idea on how to debug this?
Again, thanks for your support
Alvaro
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-697107059
Hi Otto
Thanks for your time and support
I´ve enabled RateLimitBGFetches to see what happens.
About LoadFromFile, I don´t used because, for some reason if enabled, I start to have a high rate of Caches misses (over 50%) and ressources not rewritten because of cache headers (over 75%).
About the crashes, I don´t thnk is related to shutdown since apache was not restarted at the times shown on the logs. Or do you mean another type of shitdown? Any idea on how to debug this?
Again, thanks for your support
Alvaro
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-696613828
TLDR; Based on a hunch, one thing you could try, is configuring `RateLimitBackgroundFetches` to limit the amount of in-flight work. Or switch to use `LoadFromFile` to avoid lots of in-flight http(s) fetches during shutdown altogether.
I have a hunch that this occurs during shutdown. I remember looking into something similar a long time ago, and what happens during shutdown is that some long(ish) running tasks may not return and/or respond to being cancelled in a timely manner. At that point, the module can either just force shut down, or wait longer. Waiting longer is no good, as we're mid-shutdown and that obviously should be bounded in time. So a force shutdown will be initiated, which may tear down live structures from underneath the tasks I mentioned above. Obviously, these tasks may not take that well. In this case it looks lik it the memcached implementation, which relies on apr-memcached, isn't responding (fast enough), but I'm not confident.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704596443
1.14 is a beta release. I actually don't think it's *that* likely this problem was fixed in it, but it's worth a shot.
Redis support may be in "beta" but it may not suffer from the same underlying issue that triggers the coalescing failure in memcached. The problem is -- and Lofesa pointed this out -- that the only reason we are getting into the place where mod_pagespeed crashes is that first, the connection to memcached failed. I do not know why that failed, but the underlying network connection to redis is actually a lot simpler and less likely to fail int he same way. So I think Redis is worth a shot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704592684
> Have you set `ModPagespeedMemcachedServers localhost:11211` in pagespeed config? Is the memcached instance running?
> This error `Could not find specified socket in poll list. (70015)` is related to be unable to connect to the memcached instance.
> Are tou able to do:
> `telnet localhost:11211` and run the command `stats` (for example)?
Hi Lofesa
Sure localhost:11212 is setup in pagespeed conf file and I can see pagespeed physical caches stats increasing. I can telnet to port 11212. Note that 11212 is the port of the instance dedicated to pagespeed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-696613828
TLDR; Based on a hunch, one thing you could try, is configuring `RateLimitBackgroundFetches` to limit the amount of in-flight work. Or switch to use `LoadFromFile` to avoid lots of in-flight http(s) fetches during shutdown altogether.
I have a hunch that this occurs during shutdown. I remember looking into something similar a long time ago, and what happens during shutdown is that some long(ish) running tasks may not return and/or respond to being cancelled in a timely manner. At that point, the module can either just force shut down, or wait longer. Waiting longer is no good, as we're mid-shutdown and that obviously should be bounded in time. So a force shutdown will be initiated, which may tear down live structures from underneath the tasks I mentioned above. Obviously, these tasks may not take that well. In this case it looks lik it the memcached implementation, which relies on apr-memcached, isn't responding (fast enough), but I'm not confident.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704802951
Thanks @ajconejo it is helpful to know that the problem persists with 1.14.
One more thought: I just remembered that on master I added https://github.com/apache/incubator-pagespeed-mod/commit/1cc142ecca589964348d059bc2ef8203ecd7961d
I did so after a [tsan](https://github.com/google/sanitizers) test run pointed this out as potentially problematic, but it was only caught recently on master as there we have an easy means to test with that sanitiser and others. I'm not sure this is a fix for this specific problem, but it very well might be if tsan was right about the race it observed being problematic in practice.
Unfortunately, master diverges quite a bit from 1.14, and there's no packing scripts yet to make
it low friction to build/test it, though one can produce an optimized build of mod-pagespeed's lib via:
```
bazel build -c opt libmod_pagespeed.so
```
This may take quite a while, and before doing this one needs a few build time prerequisites.
It's probably easiest to take a look at https://github.com/apache/incubator-pagespeed-mod/blob/master/.travis.yml to extract those. I would be curious if the problem persists there too. I wouldn't know how to patch in the change above using the build system for 1.14 as they diverge a lot (gyp vs bazel), hence back porting it doesn't seem straight forward to me at the moment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-699146162
Hi Otto
As I´ve two apps using memcached on the server I´ve configured a 2nd memcached instance so each one use its own.
After that, I ´m still having similar crashes. I´ll disable memcache on pagespeed and continue to monitor.
How can I get verbose logging and a crash dump for pagespeed? I´m using ubuntu bionic.
I´ve enabled RateLimitBrackgroundFetches but not yet LoadFromFile. I´m going step by step in order to pinpoint the origin of the issue.
Other thing I´ve detected is that when this happens, I´ve over a hundred connections to memcached port even though the service has very low usage. Though I didn´t had time to check the details of the process shown below.
sudo netstat -tulpan |grep 11212
tcp 0 0 127.0.0.1:11212 0.0.0.0:* LISTEN 8006/memcached
tcp 0 0 127.0.0.1:11212 127.0.0.1:46128 ESTABLISHED 8006/memcached
tcp 0 0 127.0.0.1:45876 127.0.0.1:11212 ESTABLISHED 7010/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:44842 ESTABLISHED 8006/memcached
tcp 0 0 127.0.0.1:39644 127.0.0.1:11212 ESTABLISHED 28108/apache2
tcp 0 0 127.0.0.1:45408 127.0.0.1:11212 ESTABLISHED 6173/apache2
tcp 0 0 127.0.0.1:45436 127.0.0.1:11212 ESTABLISHED 6261/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:45036 ESTABLISHED 8006/memcached
tcp 0 0 127.0.0.1:44780 127.0.0.1:11212 ESTABLISHED 5017/apache2
tcp 0 0 127.0.0.1:45904 127.0.0.1:11212 ESTABLISHED 7086/apache2
. . . .
Again, Thanks for your time. Much appreciated.
These are the crashes from today as of now.
[Fri Sep 25 14:00:38.904717 2020] [pagespeed:alert] [pid 23826] [mod_pagespeed 1.13.35.2-0 @23826] [0925/140038:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7fbc7babd70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7fbc7bab5060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7fbc7babf8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7fbc7bc58513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7fbc7bc5865b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7fbc7bbfaea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7fbc7bbfaf2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7fbc7bc5804b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7fbc7bd7d677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7fbc7bd84d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7fbc7baad02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7fbc7baad129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7fbc7bab6b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7fbc803bb0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7fbc803bb1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7fbc7cc2b9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x559566c8faf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7fbc7c81e831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7fbc7c81eb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7fbc7c81f57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x559566c692be]\n/usr/sbin/apache2(main+0x9fb) [0x559566c61b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fbc80399b97]\n/usr/sbin/apache2(_start+0x2a) [0x559566c61c5a]
[Fri Sep 25 10:14:33.275474 2020] [pagespeed:alert] [pid 4949] [mod_pagespeed 1.13.35.2-0 @4949] [0925/101433:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7fcb1f57d70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7fcb1f575060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7fcb1f57f8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7fcb1f718513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7fcb1f71865b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7fcb1f6baea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7fcb1f6baf2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7fcb1f71804b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7fcb1f83d677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7fcb1f844d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e)
[0x7fcb1f56d02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7fcb1f56d129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7fcb1f576b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7fcb23e7b0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7fcb23e7b1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7fcb206eb9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x558509bd2af0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7fcb202de831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7fcb202deb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7fcb202df57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x558509bac2be]\n/usr/sbin/apache2(main+0x9fb) [0x558509ba4b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fcb23e59b97]\n/usr/sbin/apache2(_start+0x2a) [0x558509ba4c5a]
[Fri Sep 25 08:44:21.182062 2020] [pagespeed:alert] [pid 344] [mod_pagespeed 1.13.35.2-0 @344] [0925/084421:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7fcb1f57d70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7fcb1f575060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7fcb1f57f8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7fcb1f718513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7fcb1f71865b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7fcb1f6baea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7fcb1f6baf2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7fcb1f71804b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7fcb1f83d677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7fcb1f844d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e) [0
x7fcb1f56d02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7fcb1f56d129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7fcb1f576b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7fcb23e7b0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7fcb23e7b1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7fcb206eb9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x558509bd2af0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7fcb202de831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7fcb202deb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7fcb202df57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x558509bac2be]\n/usr/sbin/apache2(main+0x9fb) [0x558509ba4b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fcb23e59b97]\n/usr/sbin/apache2(_start+0x2a) [0x558509ba4c5a]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704847541
What about this approach:
1.- Make the APR version match between 1.14 and the bazel branchs (if needed)
2.- Apply the external/apr.patch in the 1.14
3.- Build Pagespeed 1.14 with the new APR version (if needed) and the applied patch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704605313
Also, it may be worth trying a different memcached version too if that is viable and you'd like to stay on it for some reason. Switching to Redis, as @jmarantz suggested, seems like a surefire way to avoid this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704251553
Hi @jmarantz
Seems that some of these error messages, before crash, are like #522 (closed) about memcached come down in middle of running test and trow error like "AprMemCache::Put error: Could not find specified socket in poll list. on key..." and #1372 and trow similar error messages.
These both issues are that pagespeed module is unable to connect to memcached.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704645440
Hi,
I´ve upgraded to 1.14 but have similar crashes.
[Tue Oct 06 20:27:23.104906 2020] [pagespeed:warn] [pid 30364] [mod_pagespeed 1.14.36.1-0 @30364] [1006/202723:WARNING:queued_worker_pool.cc(437)] Canceling 1 functions on sequence Shutdown
[Tue Oct 06 20:29:43.620286 2020] [pagespeed:warn] [pid 30493] [mod_pagespeed 1.14.36.1-0 @30493] [1006/202943:WARNING:serf_url_async_fetcher.cc(1225)] Aborting fetch of https://www.googletagmanager.com/gtag/js?id=UA-134034004-1
[Tue Oct 06 20:29:43.624427 2020] [pagespeed:warn] [pid 30493] [mod_pagespeed 1.14.36.1-0 @30493] RateController: drop fetch of https://biblioteca.celacp.org/opac-tmpl/lib/emoji-picker/img/IconsetSmiles.png on shutdown
[Tue Oct 06 20:29:43.625274 2020] [pagespeed:warn] [pid 30493] [mod_pagespeed 1.14.36.1-0 @30493] RateController: drop fetch of https://biblioteca.celacp.org/opac-tmpl/lib/emoji-picker/img/IconsetSmiles_1x.png on shutdown
[Tue Oct 06 20:29:54.764929 2020] [pagespeed:alert] [pid 30505] [mod_pagespeed 1.14.36.1-0 @30505] [1006/202954:FATAL:cache_batcher.cc(274)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb367e) [0x7f79b6c9867e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab23d) [0x7f79b6c9023d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb5c42) [0x7f79b6c9ac42]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c4e6) [0x7f79b6e414e6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c60b) [0x7f79b6e4160b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1fa7a6) [0x7f79b6ddf7a6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25bf9b) [0x7f79b6e40f9b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x38ee77) [0x7f79b6f73e77]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x39713f) [0x7f79b6f7c13f]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2e4e) [0x7f79b6c87e4e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2f59)
[0x7f79b6c87f59]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacd25) [0x7f79b6c91d25]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f79bb5df0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f79bb5df1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f79b7e4f9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x55b171abfaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f79b7a42831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f79b7a42b34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b71) [0x7f79b7a42b71]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x425f) [0x7f79b7a4325f]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x55b171a992be]\n/usr/sbin/apache2(main+0x9fb) [0x55b171a91b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f79bb5bdb97]\n/usr/sbin/apache2(_start+0x2a) [0x55b171a91c5a]
[Tue Oct 06 20:29:54.953738 2020] [mpm_itk:error] [pid 30346] child died with signal 6
[Tue Oct 06 20:39:54.830275 2020] [pagespeed:alert] [pid 30624] [mod_pagespeed 1.14.36.1-0 @30624] [1006/203954:FATAL:cache_batcher.cc(274)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb367e) [0x7f79b6c9867e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab23d) [0x7f79b6c9023d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb5c42) [0x7f79b6c9ac42]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c4e6) [0x7f79b6e414e6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c60b) [0x7f79b6e4160b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1fa7a6) [0x7f79b6ddf7a6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25bf9b) [0x7f79b6e40f9b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x38ee77) [0x7f79b6f73e77]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x39713f) [0x7f79b6f7c13f]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2e4e) [0x7f79b6c87e4e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2f59)
[0x7f79b6c87f59]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacd25) [0x7f79b6c91d25]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f79bb5df0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f79bb5df1ea]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x32a3) [0x7f79b7a422a3]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x32bb) [0x7f79b7a422bb]\n/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f79bb99f8a0]\n/lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_timedwait+0x335) [0x7f79bb99af85]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x3b9af0) [0x7f79b6f9eaf0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x20ee7c) [0x7f79b6df3e7c]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x20ef1e) [0x7f79b6df3f1e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xd732e) [0x7f79b6cbc32e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xac7c8) [0x7f79b6c917c8]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xaeb56) [0x7f79b6c93b56]\n/usr/lib/apache2/modules/mod_filter.
so(+0x2113) [0x7f79b8469113]\n/usr/lib/apache2/modules/mod_proxy_http.so(+0x4458) [0x7f79b65b9458]\n/usr/lib/apache2/modules/mod_proxy_http.so(+0x5bbf) [0x7f79b65babbf]\n/usr/lib/apache2/modules/mod_proxy.so(proxy_run_scheme_handler+0x7c) [0x7f79b67ca37c]\n/usr/lib/apache2/modules/mod_proxy.so(+0xbf19) [0x7f79b67caf19]\n/usr/sbin/apache2(ap_run_handler+0x40) [0x55b171ab5d10]\n/usr/sbin/apache2(ap_invoke_handler+0x9d) [0x55b171ab628d]\n/usr/sbin/apache2(ap_process_async_request+0x29b) [0x55b171acdc9b]\n/usr/sbin/apache2(ap_process_request+0x10) [0x55b171acde70]\n/usr/sbin/apache2(+0x6e23d) [0x55b171aca23d]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x55b171abfaf0]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x5d) [0x7f79b7e4f9ad]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x55b171abfaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f79b7a42831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f79b7a42b34]\n/usr/lib/apache2/modules/mod
_mpm_prefork.so(+0x457e) [0x7f79b7a4357e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x55b171a992be]\n/usr/sbin/apache2(main+0x9fb) [0x55b171a91b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f79bb5bdb97]\n/usr/sbin/apache2(_start+0x2a) [0x55b171a91c5a]
[Tue Oct 06 20:39:54.854052 2020] [pagespeed:alert] [pid 31326] [mod_pagespeed 1.14.36.1-0 @31326] [1006/203954:FATAL:cache_batcher.cc(274)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb367e) [0x7f79b6c9867e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab23d) [0x7f79b6c9023d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb5c42) [0x7f79b6c9ac42]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c4e6) [0x7f79b6e414e6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25c60b) [0x7f79b6e4160b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1fa7a6) [0x7f79b6ddf7a6]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x25bf9b) [0x7f79b6e40f9b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x38ee77) [0x7f79b6f73e77]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x39713f) [0x7f79b6f7c13f]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2e4e) [0x7f79b6c87e4e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa2f59)
[0x7f79b6c87f59]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacd25) [0x7f79b6c91d25]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f79bb5df0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f79bb5df1ea]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x32a3) [0x7f79b7a422a3]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x32bb) [0x7f79b7a422bb]\n/lib/x86_64-linux-gnu/libpthread.so.0(+0x128a0) [0x7f79bb99f8a0]\n/lib/x86_64-linux-gnu/libpthread.so.0(pthread_cond_timedwait+0x335) [0x7f79bb99af85]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x3b9af0) [0x7f79b6f9eaf0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x20ee7c) [0x7f79b6df3e7c]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x20ef1e) [0x7f79b6df3f1e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xd738e) [0x7f79b6cbc38e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xac828) [0x7f79b6c91828]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xaec36) [0x7f79b6c93c36]\n/usr/lib/apache2/modules/mod_filter.
so(+0x2113) [0x7f79b8469113]\n/usr/lib/apache2/modules/mod_proxy_http.so(+0x3539) [0x7f79b65b8539]\n/usr/lib/apache2/modules/mod_proxy_http.so(+0x5bbf) [0x7f79b65babbf]\n/usr/lib/apache2/modules/mod_proxy.so(proxy_run_scheme_handler+0x7c) [0x7f79b67ca37c]\n/usr/lib/apache2/modules/mod_proxy.so(+0xbf19) [0x7f79b67caf19]\n/usr/sbin/apache2(ap_run_handler+0x40) [0x55b171ab5d10]\n/usr/sbin/apache2(ap_invoke_handler+0x9d) [0x55b171ab628d]\n/usr/sbin/apache2(ap_process_async_request+0x29b) [0x55b171acdc9b]\n/usr/sbin/apache2(ap_process_request+0x10) [0x55b171acde70]\n/usr/sbin/apache2(+0x6e23d) [0x55b171aca23d]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x55b171abfaf0]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x5d) [0x7f79b7e4f9ad]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x55b171abfaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f79b7a42831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f79b7a42b34]\n/usr/lib/apache2/modules/mod
_mpm_prefork.so(+0x3b71) [0x7f79b7a42b71]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x425f) [0x7f79b7a4325f]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x55b171a992be]\n/usr/sbin/apache2(main+0x9fb) [0x55b171a91b5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f79bb5bdb97]\n/usr/sbin/apache2(_start+0x2a) [0x55b171a91c5a]
[Tue Oct 06 20:39:55.139495 2020] [mpm_prefork:notice] [pid 30341] AH00169: caught SIGTERM, shutting down
[Tue Oct 06 20:42:03.572693 2020] [mpm_prefork:notice] [pid 32377] AH00163: Apache/2.4.29 (Ubuntu) mpm-itk/2.4.7-04 OpenSSL/1.1.1 configured -- resuming normal operations
[Tue Oct 06 20:42:03.575089 2020] [core:notice] [pid 32377] AH00094: Command line: '/usr/sbin/apache2'
[Tue Oct 06 20:42:50.780584 2020] [pagespeed:warn] [pid 32439] [mod_pagespeed 1.14.36.1-0 @32439] [1006/204250:WARNING:queued_worker_pool.cc(437)] Canceling 1 functions on sequence Shutdown
This time I was able to get some netstat info during/after crash. Note thar 11211 correspond to the other memcached instance. the process running on it is not affected by the pagespeed crash
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11211
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52948 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52630 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:52642 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52632 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:52948 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52966 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52946 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:46492 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52632 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52946 ESTABLISHED 22360/memcached
tcp 1 0 127.0.0.1:50238 127.0.0.1:11211 CLOSE_WAIT 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52966 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:46490 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52642 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:46490 127.0.0.1:11211 ESTABLISHED 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52630 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:46492 127.0.0.1:11211 ESTABLISHED 1470/perl
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11212
tcp 0 0 127.0.0.1:11212 0.0.0.0:* LISTEN 22374/memcached
tcp 0 0 127.0.0.1:11212 127.0.0.1:57962 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57954 127.0.0.1:11212 CLOSE_WAIT 30586/apache2
tcp 1 0 127.0.0.1:57970 127.0.0.1:11212 CLOSE_WAIT 30624/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57976 ESTABLISHED 22374/memcached
tcp 1 0 127.0.0.1:57942 127.0.0.1:11212 CLOSE_WAIT 30529/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57964 FIN_WAIT2 -
tcp 0 0 127.0.0.1:57976 127.0.0.1:11212 ESTABLISHED 30642/apache2
tcp 1 0 127.0.0.1:57950 127.0.0.1:11212 CLOSE_WAIT 30571/apache2
tcp 1 0 127.0.0.1:57946 127.0.0.1:11212 CLOSE_WAIT 30535/apache2
tcp 1 0 127.0.0.1:57962 127.0.0.1:11212 CLOSE_WAIT 30612/apache2
tcp 1 0 127.0.0.1:57964 127.0.0.1:11212 CLOSE_WAIT 30619/apache2
tcp 1 0 127.0.0.1:57948 127.0.0.1:11212 CLOSE_WAIT 30555/apache2
tcp 1 0 127.0.0.1:57960 127.0.0.1:11212 CLOSE_WAIT 30607/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57970 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57956 127.0.0.1:11212 CLOSE_WAIT 30599/apache2
acornejo@Biblio:~$ ps -ef |grep 30642
bibliot+ 30642 30590 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 30656 30111 0 20:33 pts/2 00:00:00 grep --color=auto 30642
acornejo@Biblio:~$ ps -ef |grep 30571
bibliot+ 30571 30348 0 20:31 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 30664 30111 0 20:34 pts/2 00:00:00 grep --color=auto 30571
acornejo@Biblio:~$ ps -ef |grep 30607
bibliot+ 30607 30539 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 30668 30111 0 20:34 pts/2 00:00:00 grep --color=auto 30607
acornejo@Biblio:~$ ps -ef |grep 22374
memcache 22374 1 0 Oct05 ? 00:01:27 /usr/bin/memcached -o idle_timeout=20 -v -m 256 -p 11212 -u memcache -l 127.0.0.1 -r -P /var/run/memcached/memcached_server2.pid -t 6
acornejo 31322 30111 0 20:34 pts/2 00:00:00 grep --color=auto 22374
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11212
tcp 0 0 127.0.0.1:11212 0.0.0.0:* LISTEN 22374/memcached
tcp 1 0 127.0.0.1:57954 127.0.0.1:11212 CLOSE_WAIT 30586/apache2
tcp 1 0 127.0.0.1:57970 127.0.0.1:11212 CLOSE_WAIT 30624/apache2
tcp 1 0 127.0.0.1:57982 127.0.0.1:11212 CLOSE_WAIT 31271/apache2
tcp 1 0 127.0.0.1:57942 127.0.0.1:11212 CLOSE_WAIT 30529/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57982 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57976 127.0.0.1:11212 CLOSE_WAIT 30642/apache2
tcp 1 0 127.0.0.1:57950 127.0.0.1:11212 CLOSE_WAIT 30571/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57978 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57946 127.0.0.1:11212 CLOSE_WAIT 30535/apache2
tcp 1 0 127.0.0.1:57962 127.0.0.1:11212 CLOSE_WAIT 30612/apache2
tcp 1 0 127.0.0.1:57964 127.0.0.1:11212 CLOSE_WAIT 30619/apache2
tcp 1 0 127.0.0.1:57948 127.0.0.1:11212 CLOSE_WAIT 30555/apache2
tcp 1 0 127.0.0.1:57960 127.0.0.1:11212 CLOSE_WAIT 30607/apache2
tcp 1 0 127.0.0.1:57978 127.0.0.1:11212 CLOSE_WAIT 30671/apache2
tcp 1 0 127.0.0.1:57956 127.0.0.1:11212 CLOSE_WAIT 30599/apache2
acornejo@Biblio:~$ ps -ef |grep apache
root 29798 29719 0 20:21 pts/0 00:00:00 sudo tail -f /var/log/apache2/error.log
root 29799 29798 0 20:21 pts/0 00:00:00 tail -f /var/log/apache2/error.log
root 30341 1 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30342 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30343 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30344 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30345 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30348 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30349 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30350 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30351 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30352 30341 0 20:26 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30365 30341 0 20:27 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30408 30341 0 20:28 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30419 30341 0 20:28 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30518 30344 0 20:29 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30529 30419 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30534 30341 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30535 30350 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30539 30341 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30555 30345 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30559 30342 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30561 30341 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30562 30341 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30563 30341 0 20:30 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30571 30348 0 20:31 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30586 30343 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30590 30341 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30599 30534 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30602 30341 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30607 30539 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30610 30341 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30612 30561 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30618 30341 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30619 30563 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30623 30341 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30624 30408 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30625 30341 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30642 30590 0 20:33 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 30671 30365 0 20:34 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 30673 30341 0 20:34 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31271 30610 0 20:34 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31320 30341 0 20:34 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31326 30351 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31328 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31334 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31335 30625 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31337 30349 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31339 30562 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31343 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31344 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31452 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31453 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31454 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 31455 30341 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31783 30602 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
bibliot+ 31964 30673 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 32003 31328 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 32005 30111 0 20:35 pts/2 00:00:00 grep --color=auto apache
acornejo@Biblio:~$ ps -ef |grep apache |wc
59 591 4444
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11212
tcp 0 0 127.0.0.1:11212 0.0.0.0:* LISTEN 22374/memcached
tcp 1 0 127.0.0.1:58002 127.0.0.1:11212 CLOSE_WAIT 31964/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57992 FIN_WAIT2 -
tcp 0 0 127.0.0.1:11212 127.0.0.1:58002 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57954 127.0.0.1:11212 CLOSE_WAIT 30586/apache2
tcp 1 0 127.0.0.1:57970 127.0.0.1:11212 CLOSE_WAIT 30624/apache2
tcp 1 0 127.0.0.1:57982 127.0.0.1:11212 CLOSE_WAIT 31271/apache2
tcp 1 0 127.0.0.1:57942 127.0.0.1:11212 CLOSE_WAIT 30529/apache2
tcp 1 0 127.0.0.1:57998 127.0.0.1:11212 CLOSE_WAIT 31339/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57996 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57976 127.0.0.1:11212 CLOSE_WAIT 30642/apache2
tcp 1 0 127.0.0.1:57996 127.0.0.1:11212 CLOSE_WAIT 31337/apache2
tcp 1 0 127.0.0.1:57950 127.0.0.1:11212 CLOSE_WAIT 30571/apache2
tcp 1 0 127.0.0.1:57992 127.0.0.1:11212 CLOSE_WAIT 31335/apache2
tcp 1 0 127.0.0.1:57946 127.0.0.1:11212 CLOSE_WAIT 30535/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57998 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57986 127.0.0.1:11212 CLOSE_WAIT 31326/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:58004 ESTABLISHED 22374/memcached
tcp 1 0 127.0.0.1:57962 127.0.0.1:11212 CLOSE_WAIT 30612/apache2
tcp 1 0 127.0.0.1:57964 127.0.0.1:11212 CLOSE_WAIT 30619/apache2
tcp 0 0 127.0.0.1:58004 127.0.0.1:11212 ESTABLISHED 32003/apache2
tcp 1 0 127.0.0.1:57948 127.0.0.1:11212 CLOSE_WAIT 30555/apache2
tcp 1 0 127.0.0.1:57960 127.0.0.1:11212 CLOSE_WAIT 30607/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:57986 FIN_WAIT2 -
tcp 1 0 127.0.0.1:57978 127.0.0.1:11212 CLOSE_WAIT 30671/apache2
tcp 1 0 127.0.0.1:57956 127.0.0.1:11212 CLOSE_WAIT 30599/apache2
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11211
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52948 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52630 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:52642 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52632 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:52948 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52966 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52946 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:46492 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52632 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52946 ESTABLISHED 22360/memcached
tcp 1 0 127.0.0.1:50238 127.0.0.1:11211 CLOSE_WAIT 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52966 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:46490 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52642 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:46490 127.0.0.1:11211 ESTABLISHED 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52630 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:46492 127.0.0.1:11211 ESTABLISHED 1470/perl
acornejo@Biblio:~$ sudo netstat -tulpan |grep apache
tcp 1 0 127.0.0.1:58002 127.0.0.1:11212 CLOSE_WAIT 31964/apache2
tcp 1 0 127.0.0.1:57954 127.0.0.1:11212 CLOSE_WAIT 30586/apache2
tcp 1 0 127.0.0.1:57970 127.0.0.1:11212 CLOSE_WAIT 30624/apache2
tcp 1 0 127.0.0.1:57982 127.0.0.1:11212 CLOSE_WAIT 31271/apache2
tcp 1 0 127.0.0.1:57942 127.0.0.1:11212 CLOSE_WAIT 30529/apache2
tcp 1 0 127.0.0.1:57998 127.0.0.1:11212 CLOSE_WAIT 31339/apache2
tcp 0 0 127.0.0.1:58008 127.0.0.1:11212 ESTABLISHED 32027/apache2
tcp 1 0 127.0.0.1:57976 127.0.0.1:11212 CLOSE_WAIT 30642/apache2
tcp 1 0 127.0.0.1:57996 127.0.0.1:11212 CLOSE_WAIT 31337/apache2
tcp 1 0 127.0.0.1:57950 127.0.0.1:11212 CLOSE_WAIT 30571/apache2
tcp 1 0 127.0.0.1:57992 127.0.0.1:11212 CLOSE_WAIT 31335/apache2
tcp 1 0 127.0.0.1:57946 127.0.0.1:11212 CLOSE_WAIT 30535/apache2
tcp 1 0 127.0.0.1:57986 127.0.0.1:11212 CLOSE_WAIT 31326/apache2
tcp 0 0 127.0.0.1:58012 127.0.0.1:11212 ESTABLISHED 32030/apache2
tcp 1 0 127.0.0.1:57962 127.0.0.1:11212 CLOSE_WAIT 30612/apache2
tcp 1 0 127.0.0.1:57964 127.0.0.1:11212 CLOSE_WAIT 30619/apache2
tcp 1 0 127.0.0.1:58004 127.0.0.1:11212 CLOSE_WAIT 32003/apache2
tcp 1 0 127.0.0.1:57948 127.0.0.1:11212 CLOSE_WAIT 30555/apache2
tcp 1 0 127.0.0.1:57960 127.0.0.1:11212 CLOSE_WAIT 30607/apache2
tcp 1 0 127.0.0.1:57978 127.0.0.1:11212 CLOSE_WAIT 30671/apache2
tcp 1 0 127.0.0.1:57956 127.0.0.1:11212 CLOSE_WAIT 30599/apache2
tcp6 0 0 :::443 :::* LISTEN 30341/apache2
tcp6 0 0 :::8080 :::* LISTEN 30341/apache2
tcp6 0 0 :::80 :::* LISTEN 30341/apache2
tcp6 32 0 45.79.198.124:443 114.119.156.95:1860 CLOSE_WAIT 30348/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51288 CLOSE_WAIT 30561/apache2
tcp6 0 0 45.79.198.124:443 66.249.79.109:56234 ESTABLISHED 30625/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51280 CLOSE_WAIT 30344/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51292 CLOSE_WAIT 30408/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51287 CLOSE_WAIT 30539/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.18:56548 CLOSE_WAIT 30365/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51285 CLOSE_WAIT 30343/apache2
tcp6 1 0 45.79.198.124:443 66.249.79.109:43552 CLOSE_WAIT 30419/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51297 CLOSE_WAIT 30602/apache2
tcp6 32 0 45.79.198.124:443 114.119.132.250:24044 CLOSE_WAIT 30590/apache2
tcp6 0 0 45.79.198.124:443 185.191.171.13:53354 ESTABLISHED 31343/apache2
tcp6 0 0 45.79.198.124:443 114.119.145.47:43566 ESTABLISHED 31328/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.16:41944 CLOSE_WAIT 30563/apache2
tcp6 1 0 45.79.198.124:443 195.246.120.138:57041 CLOSE_WAIT 30610/apache2
tcp6 0 0 45.79.198.124:443 201.230.13.16:51294 ESTABLISHED 30351/apache2
tcp6 0 0 45.79.198.124:443 201.230.13.16:51295 ESTABLISHED 30562/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.18:43264 CLOSE_WAIT 30349/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.16:21588 CLOSE_WAIT 30534/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.1:36590 CLOSE_WAIT 30350/apache2
tcp6 1 0 45.79.198.124:443 201.230.13.16:51282 CLOSE_WAIT 30342/apache2
tcp6 1 0 45.79.198.124:443 185.191.171.1:6840 CLOSE_WAIT 30345/apache2
tcp6 0 0 45.79.198.124:443 201.230.13.16:51298 ESTABLISHED 30673/apache2
tcp6 0 0 45.79.198.124:443 195.246.120.138:59970 ESTABLISHED 31344/apache2
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11211
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52948 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52630 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:52642 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52632 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:52948 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52966 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:52946 127.0.0.1:11211 ESTABLISHED 30476/starman worke
tcp 0 0 127.0.0.1:11211 127.0.0.1:46492 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52632 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:52946 ESTABLISHED 22360/memcached
tcp 1 0 127.0.0.1:50238 127.0.0.1:11211 CLOSE_WAIT 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52966 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:11211 127.0.0.1:46490 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:52642 127.0.0.1:11211 ESTABLISHED 29567/starman worke
tcp 0 0 127.0.0.1:46490 127.0.0.1:11211 ESTABLISHED 1470/perl
tcp 0 0 127.0.0.1:11211 127.0.0.1:52630 ESTABLISHED 22360/memcached
tcp 0 0 127.0.0.1:46492 127.0.0.1:11211 ESTABLISHED 1470/perl
acornejo@Biblio:~$ sudo netstat -tulpan |grep 11212
tcp 0 0 127.0.0.1:11212 0.0.0.0:* LISTEN 22374/memcached
tcp 0 0 127.0.0.1:11212 127.0.0.1:58008 FIN_WAIT2 -
tcp 1 0 127.0.0.1:58002 127.0.0.1:11212 CLOSE_WAIT 31964/apache2
tcp 1 0 127.0.0.1:57954 127.0.0.1:11212 CLOSE_WAIT 30586/apache2
tcp 1 0 127.0.0.1:57970 127.0.0.1:11212 CLOSE_WAIT 30624/apache2
tcp 1 0 127.0.0.1:57982 127.0.0.1:11212 CLOSE_WAIT 31271/apache2
tcp 1 0 127.0.0.1:57942 127.0.0.1:11212 CLOSE_WAIT 30529/apache2
tcp 1 0 127.0.0.1:57998 127.0.0.1:11212 CLOSE_WAIT 31339/apache2
tcp 0 0 127.0.0.1:11212 127.0.0.1:58012 FIN_WAIT2 -
tcp 1 0 127.0.0.1:58008 127.0.0.1:11212 CLOSE_WAIT 32027/apache2
tcp 1 0 127.0.0.1:57976 127.0.0.1:11212 CLOSE_WAIT 30642/apache2
tcp 1 0 127.0.0.1:57996 127.0.0.1:11212 CLOSE_WAIT 31337/apache2
tcp 1 0 127.0.0.1:57950 127.0.0.1:11212 CLOSE_WAIT 30571/apache2
tcp 1 0 127.0.0.1:57992 127.0.0.1:11212 CLOSE_WAIT 31335/apache2
tcp 1 0 127.0.0.1:57946 127.0.0.1:11212 CLOSE_WAIT 30535/apache2
tcp 1 0 127.0.0.1:57986 127.0.0.1:11212 CLOSE_WAIT 31326/apache2
tcp 1 0 127.0.0.1:58012 127.0.0.1:11212 CLOSE_WAIT 32030/apache2
tcp 1 0 127.0.0.1:57962 127.0.0.1:11212 CLOSE_WAIT 30612/apache2
tcp 1 0 127.0.0.1:57964 127.0.0.1:11212 CLOSE_WAIT 30619/apache2
tcp 1 0 127.0.0.1:58004 127.0.0.1:11212 CLOSE_WAIT 32003/apache2
tcp 1 0 127.0.0.1:57948 127.0.0.1:11212 CLOSE_WAIT 30555/apache2
tcp 1 0 127.0.0.1:57960 127.0.0.1:11212 CLOSE_WAIT 30607/apache2
tcp 1 0 127.0.0.1:57978 127.0.0.1:11212 CLOSE_WAIT 30671/apache2
tcp 1 0 127.0.0.1:57956 127.0.0.1:11212 CLOSE_WAIT 30599/apache2
acornejo@Biblio:~$ sudo ps -ef |grep 30599
bibliot+ 30599 30534 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 32050 30111 0 20:37 pts/2 00:00:00 grep --color=auto 30599
acornejo@Biblio:~$ sudo ps -ef |grep 30612
bibliot+ 30612 30561 0 20:32 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 32067 30111 0 20:38 pts/2 00:00:00 grep --color=auto 30612
acornejo@Biblio:~$ sudo ps -ef |grep 31964
bibliot+ 31964 30673 0 20:35 ? 00:00:00 /usr/sbin/apache2 -k start
acornejo 32081 30111 0 20:38 pts/2 00:00:00 grep --color=auto 31964
And this is my physical cache stats just before the crash.
I also noted that after the crash, pagespeed admin page stop working. I guess is related to the fact that apache is somehow compromised by the pagespeed crash.
Will try redis for the moment and see what happens
Regards and thanks to all
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-699479202
Have you tried to take out memcached?. For test purpose, maybe the problem is in the memcached side.
My knowledge about memcached is not too much, but if you have hundred of conections to memcached, maybe, it´s because memcached have hundred of clients or memcached don´t close idle connections.
In both cases maybe is reaching the max number of connections (I read 1024 by default) and this can cause timeouts and problems.
Memcached had a start parameter `idle_timeout` that close the conection when idle for the time set in it in seconds. By default is disabled. In the script that start memcached you must see some like `-o idle_timeout=X` where X is the idle time in seconds.
For this parameter memcached version must be at least 1.4.33.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-697233225
Well, the `[0921/150340:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight.end().` in the logs might be a hint, or a red herring (a crash elsewhere might cause false flags). Starting out with that:
- If the message above is what starts the crash, turning off memcached should put a stop to that, as I think that is the only consumer of the code involved. So if we can try this, and the crashes go away, we've pinpointed a general area to look at.
- It might also help to turn on the most verbose logging, to hopefully get a chance at seeing what happens right before the crash. This could help pinpoint if there's a specific timing to this (e.g. shutdown, worker recycle, etc).
- Lastly, a crash dump might be useful as that allows inspection with gdb after the fact. It would be useful to know what all threads are at when this happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] ajconejo commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
ajconejo commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-703948621
Hi
After a week running mod_pagespeed without memcached with no issues I decided to reenable it.
Unfortunaly little later, pagespeed started to crash with:
[Mon Oct 05 08:41:09.379460 2020] [pagespeed:alert] [pid 12407] [mod_pagespeed 1.13.35.2-0 @12407] [1005/084109:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x49d3) [0x7f48f32fe9d3]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 10:19:07.680729 2020] [pagespeed:alert] [pid 18613] [mod_pagespeed 1.13.35.2-0 @18613] [1005/101907:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7f48f32fe57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 10:20:12.539184 2020] [pagespeed:alert] [pid 18656] [mod_pagespeed 1.13.35.2-0 @18656] [1005/102012:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x49d3) [0x7f48f32fe9d3]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 10:51:32.493247 2020] [pagespeed:alert] [pid 20910] [mod_pagespeed 1.13.35.2-0 @20910] [1005/105132:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x49d3) [0x7f48f32fe9d3]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 11:50:45.268220 2020] [pagespeed:alert] [pid 25535] [mod_pagespeed 1.13.35.2-0 @25535] [1005/115045:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x49d3) [0x7f48f32fe9d3]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 11:55:14.183452 2020] [pagespeed:alert] [pid 25732] [mod_pagespeed 1.13.35.2-0 @25732] [1005/115514:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x49d3) [0x7f48f32fe9d3]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
[Mon Oct 05 12:17:34.643193 2020] [pagespeed:alert] [pid 27189] [mod_pagespeed 1.13.35.2-0 @27189] [1005/121734:FATAL:cache_batcher.cc(272)] Check failed: iter != in_flight_.end(). \n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb370e) [0x7f48f259c70e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xab060) [0x7f48f2594060]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xb58e0) [0x7f48f259e8e0]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e513) [0x7f48f2737513]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e65b) [0x7f48f273765b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0ea4) [0x7f48f26d9ea4]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x1f0f2d) [0x7f48f26d9f2d]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x24e04b) [0x7f48f273704b]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x373677) [0x7f48f285c677]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0x37ad9a) [0x7f48f2863d9a]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa302e
) [0x7f48f258c02e]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xa3129) [0x7f48f258c129]\n/usr/lib/apache2/modules/mod_pagespeed_ap24.so(+0xacb55) [0x7f48f2595b55]\n/lib/x86_64-linux-gnu/libc.so.6(+0x430f1) [0x7f48f6e9a0f1]\n/lib/x86_64-linux-gnu/libc.so.6(+0x431ea) [0x7f48f6e9a1ea]\n/usr/lib/apache2/modules/mpm_itk.so(itk_fork_process+0x6c) [0x7f48f370a9bc]\n/usr/sbin/apache2(ap_run_process_connection+0x40) [0x5617564ebaf0]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3831) [0x7f48f32fd831]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x3b34) [0x7f48f32fdb34]\n/usr/lib/apache2/modules/mod_mpm_prefork.so(+0x457e) [0x7f48f32fe57e]\n/usr/sbin/apache2(ap_run_mpm+0x4e) [0x5617564c52be]\n/usr/sbin/apache2(main+0x9fb) [0x5617564bdb5b]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48f6e78b97]\n/usr/sbin/apache2(_start+0x2a) [0x5617564bdc5a]
I´ve also detected several AprMemCache errors; however, seems not directly related to craches since times does not correlate.:
[Mon Oct 05 10:08:15.813077 2020] [pagespeed:error] [pid 18121] [mod_pagespeed 1.13.35.2-0 @18121] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca.celacp.org/opac-tmpl,_bootstrap,_lib,_jquery,_plugins,_jquery.highlight-3_19.1109000.js+opac-tmpl,_lib,_kjua,_kjua.min_19.1109000.js+cgi-bin,_koha,__,Mjo.48CVbwaxQy.js+cgi-bin,_koha,__,Mjo.y7bty-p3y0.js.pagespeed.jc.Lvi148d4IH.js, value-size 14373
[Mon Oct 05 10:08:25.743756 2020] [pagespeed:error] [pid 18121] [mod_pagespeed 1.13.35.2-0 @18121] AprMemCache::Put error: Broken pipe (32) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-detail.pl?biblionumber=205968_JlnJS-qx8-@Desktop@dependencies_cohort, value-size 681
[Mon Oct 05 10:08:25.752195 2020] [pagespeed:error] [pid 18121] [mod_pagespeed 1.13.35.2-0 @18121] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key v3/celacp.org/https://biblioteca.celacp.org/opac-tmpl,_bootstrap,_lib,_jquery,_plugins,_jquery.highlight-3_19.1109000.js+opac-tmpl,_lib,_kjua,_kjua.min_19.1109000.js+cgi-bin,_koha,__,Mjo.qPW0IeFMTc.js+cgi-bin,_koha,__,Mjo.y7bty-p3y0.js.pagespeed.jc.AieqmGoKxa.js, value-size 14357
[Mon Oct 05 10:08:25.754207 2020] [pagespeed:error] [pid 18121] [mod_pagespeed 1.13.35.2-0 @18121] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key rname/jc_JlnJS-qx8-3ij3qSYEsp/8/Oo7EeyqRxHJLDSvoM5C@v4, value-size 499
[Mon Oct 05 10:25:16.132021 2020] [pagespeed:error] [pid 18815] [mod_pagespeed 1.13.35.2-0 @18815] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca.celacp.org/opac-tmpl,_lib,_emoji-picker,_js,_jquery.emojiarea_19.1109000.js+opac-tmpl,_lib,_emoji-picker,_js,_emoji-picker_19.1109000.js+cgi-bin,_koha,__,Mjo.N5AEEkb8xb.js+opac-tmpl,_bootstrap,_js,_basket_19.1109000.js+opac-tmpl,_bootstrap,_js,_tags_19.1109000.js+opac-tmpl,_bootstrap,_js,_google-jackets_19.1109000.js.pagespeed.jc.sbAEOB8rq7.js, value-size 10595
[Mon Oct 05 10:25:54.878229 2020] [pagespeed:error] [pid 18815] [mod_pagespeed 1.13.35.2-0 @18815] AprMemCache::Put error: Broken pipe (32) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-detail.pl?biblionumber=168011_JlnJS-qx8-@Desktop@dependencies_cohort, value-size 681
[Mon Oct 05 10:25:54.885805 2020] [pagespeed:error] [pid 18815] [mod_pagespeed 1.13.35.2-0 @18815] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key v3/celacp.org/https://biblioteca.celacp.org/opac-tmpl,_bootstrap,_lib,_jquery,_plugins,_jquery.highlight-3_19.1109000.js+opac-tmpl,_lib,_kjua,_kjua.min_19.1109000.js+cgi-bin,_koha,__,Mjo.D4FH3lrQAH.js+cgi-bin,_koha,__,Mjo.y7bty-p3y0.js.pagespeed.jc.pxGGzKqKT_.js, value-size 14367
[Mon Oct 05 10:25:54.892019 2020] [pagespeed:error] [pid 18815] [mod_pagespeed 1.13.35.2-0 @18815] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key rname/jc_JlnJS-qx8-3ij3qSYEsp/Z/iNDFd4tVNnKYfObJERz@v4, value-size 497
[Mon Oct 05 10:25:54.893778 2020] [pagespeed:error] [pid 18815] [mod_pagespeed 1.13.35.2-0 @18815] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key rname/jc_JlnJS-qx8-3ij3qSYEsp/B/k0zJ3WtuJZ-NjHAceoy@v4, value-size 395
[Mon Oct 05 10:56:53.394302 2020] [pagespeed:error] [pid 22044] [mod_pagespeed 1.13.35.2-0 @22044] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js, value-size 901755
[Mon Oct 05 10:57:12.494172 2020] [pagespeed:error] [pid 22067] [mod_pagespeed 1.13.35.2-0 @22067] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js, value-size 901755
[Mon Oct 05 10:57:12.874348 2020] [pagespeed:error] [pid 22067] [mod_pagespeed 1.13.35.2-0 @22067] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js.pagespeed.jm.gahYYWZ2gb.js, value-size 899910
[Mon Oct 05 10:57:13.349259 2020] [pagespeed:error] [pid 22067] [mod_pagespeed 1.13.35.2-0 @22067] AprMemCache::Put error: Internal error (specific information not available) (20014) on key v3/celacp.org/https://biblioteca-admin.celacp.org/intranet-tmpl/lib/datatables/datatables.min_19.1109000.js, value-size 901736
[Mon Oct 05 12:32:04.735766 2020] [pagespeed:error] [pid 29054] [mod_pagespeed 1.13.35.2-0 @29054] AprMemCache::Get error: Internal error (specific information not available) (20014) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-search.pl?q=augusto+de+campos&branch_group_limit=_JlnJS-qx8-@Desktop@beacon_cohort
[Mon Oct 05 12:32:04.735873 2020] [pagespeed:error] [pid 29054] [mod_pagespeed 1.13.35.2-0 @29054] AprMemCache::Get error: Broken pipe (32) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-search.pl?q=augusto+de+campos&branch_group_limit=_JlnJS-qx8-@Desktop@dom
[Mon Oct 05 12:32:04.736215 2020] [pagespeed:error] [pid 29054] [mod_pagespeed 1.13.35.2-0 @29054] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-search.pl?q=augusto+de+campos&branch_group_limit=_JlnJS-qx8-@Desktop@beacon_cohort, value-size 70
[Mon Oct 05 12:32:04.750385 2020] [pagespeed:error] [pid 29054] [mod_pagespeed 1.13.35.2-0 @29054] AprMemCache::Put error: Could not find specified socket in poll list. (70015) on key prop_page/https://biblioteca.celacp.org/cgi-bin/koha/opac-search.pl?q=augusto+de+campos&branch_group_limit=_JlnJS-qx8-@Desktop@beacon_cohort, value-size 185
acornejo@Biblio:~$
I´m running stock memcached 1.5.6 (ubuntu 18.04) under Apache/2.4.29 in an exclusive instance for pagespeed. The other memcached instance runs smoothly with no issues.
As I´m using mpm_prefork, I defined 2 threads per type to see if the issue comes from an overload but had no effect
ModPagespeedNumRewriteThreads 2
ModPagespeedNumExpensiveRewriteThreads 2
BTW, a non related question. Is it possible to have the console show more than 24hrs of data?
This is my pagespeed config
Version: 14: on
Filters
ah Add Head
ij Canonicalize Javascript library URLs
cc Combine Css
jc Combine Javascript
gp Convert Gif to Png
jp Convert Jpeg to Progressive
jw Convert Jpeg To Webp
mc Convert Meta Tags
pj Convert Png to Jpeg
ws When converting images to WebP, prefer lossless conversions
dd Dedup Inlined Images
di Delay Images
ec Cache Extend Css
ei Cache Extend Images
es Cache Extend Scripts
fc Fallback Rewrite Css
if Flatten CSS Imports
hpsr Hint Preload of Subresources
hw Flushes html
ci Inline Css
gf Inline Google Font CSS
ii Inline Images
il Inline @import to Link
ji Inline Javascript
io In-place optimize for browser
idp Insert DNS Prefetch
id Insert Image Dimensions
js Jpeg Subsampling
co Outline Css
jo Outline Javascript
rj Recompress Jpeg
rp Recompress Png
rw Recompress Webp
rc Remove Comments
ri Resize Images
rm Resize Mobile Images
cf Rewrite Css
jm Rewrite External Javascript
jj Rewrite Inline Javascript
cu Rewrite Style Attributes With Url
cp Strip Image Color Profiles
md Strip Image Meta Data
Options
CacheFlushPollIntervalSec (acfpi) 1296000
CssOutlineMinBytes (co) 3000
EnableCachePurge (euci) True
EnableRewriting (e) 1
FetchHttps (fhs) enable
FetchWithGzip (afg) True
FileCacheCleanIntervalMs (afcci) 1296000000
FileCacheInodeLimit (afcl) 5000000
FileCachePath (afcp) /var/cache/mod_pagespeed/
FileCacheSizeKb (afc) 2048000
GoogleFontCssInlineMaxBytes (gfci) 100000
HttpCacheCompressionLevel (hccl) 9
ImageMaxRewritesAtOnce (im) -1
InPlaceResourceOptimization (ipro) True
JsOutlineMinBytes (jo) 3000
LoadFromFileCacheTtlMs (lfct) 4000000000
LogDir (ald) /var/log/pagespeed
MaxCacheableContentLength (rcl) 20000000
MaxCombinedCssBytes (xcc) 1024000
MaxCombinedJsBytes (xcj) 1024000
PurgeMethod (pm) PURGE
RewriteRandomDropPercentage (rrdp) -1
ShmMetadataCacheCheckpointIntervalSec (smci) 300
SslCertDirectory (assld) /etc/ssl/certs
StatisticsLogging (asle) True
StatisticsLoggingIntervalMs (asli) 600000
StatisticsLoggingMaxFileSizeKb (aslfs) 10240
Domain Lawyer
http*://*.googletagmanager.com/ Auth
http*://*.ssl-images-amazon.com/ Auth
http*://45.79.198.124/ Auth
http*://biblioteca-admin.celacp.org/ Auth
http*://biblioteca.celacp.org/ Auth
Invalidation Timestamp: (none)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] oschaaf commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
oschaaf commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704604371
I think we did upgrade apr/aprutil in 1.14, so it's worth a shot. I'm not super confident either that it will fix the problem, but I can't say there's 0% chance either.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2041: mod_pagespeed crashes from time to time
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2041:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2041#issuecomment-704206060
Hi -- thanks for the stack-traces. I do not remember seeing this problem before, but would you be willing to try 2 things:
1. upgrade to mod_pagespeed 1.14 to see if this resolves the problem
2. switch from memcached to redis
I can't promise either of these fixes the issue, but it would be good at least to confirm whether (a) this was fixed already in 1.14 and (b) whether this is specific to memcached or common to the redis/memcache implementations. If it is memcached-specific then you might be happy with Redis as an alternative anyway, and in either case we'll know more once you switch.
In case that does not work, I've looked at the code, and this was a feature added to coalesce multiple asynchronous reads of the same key. I think in retrospect this should've been made an option in case something like this went awry. It might be possible to disable or workaround this code, which was added 4 years ago in this commit: https://github.com/apache/incubator-pagespeed-mod/commit/42098f8ed9398ef1ce37a337c2aee2bdd61ee946 .
Ideally if anyone has time and wants a C++ challenge, trying to repro and fix this would be a great contribution!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org