You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pagespeed.apache.org by GitBox <gi...@apache.org> on 2022/04/11 19:35:08 UTC
[GitHub] [incubator-pagespeed-mod] Ungov77 opened a new issue, #2089: High TTFB with PageSpeed
Ungov77 opened a new issue, #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089
Hi,
TTFB is much higher when PageSpeed is enabled and I don't understand why
264ms TTFB with PageSpeed=off
https://speedvitals.com/report/www.keinegotterkeinemeister.de/eEuqEG/
1100ms TTFB with PageSpeed ON
https://speedvitals.com/report/www.keinegotterkeinemeister.de/tg1zwa/
Pagespeed config:
https://pastebin.com/fLqVcP1M
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1097268729
I see this error in /mod_pagespeed_statistics?cache#physical_cache
`Error getting memcached server status for localhost:11211`
I temporarily re-enabled public statistics on both domains if you want to check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1097225781
Thanks, we disabled defer_javascript and some other filters
However today we noticed that images are not getting converted to webp anymore after switching to Memcached
When commenting out Memcached in the config, the images are converted again to webp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1098084824
It's possible the reason that TTFB was good for a while is that the memcached code is good at failing-fast if it looks like memcached is not working.
I can think of three reasons generally that PageSpeed would make TTFB noticeably worse:
* HTML parsing overhead: Running with ?PageSpeedFilters=+debug shows us that parsing overhead is less than 6ms so I don't think it's that.
* Rewrite Deadlines: in the config you pasted above it looks like that's set to 20ms which should be fine. But it might be worth looking at the config again to see if anything deadline-related is overridden
* General overhead and queuing on your server. PageSpeed keeps a limited number of threads for each process, with work queues for each thread, and at some scale the queue-depths might start growing. There are histograms for that on /pagespeed_admin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095809062
Hi, thanks for the reply
The server is a dedicated server in OVH Datacenter in Beauharnois, Quebec, Canada
Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz (16 core(s)) - 2499 MHz
64 GB Ram
NVME SSD
1GBPS network
CPU load is always under 25% https://prnt.sc/Xte_RHbEGSgR
We also integrated a benchmark into the website to calculate the page load speed of the different PHP modules and calculate SQL queries execution time. You can see it on the bottom of the page when adding ?benchmark=1 to the URL
https://www.keinegotterkeinemeister.de/tshirt-sabotage-black-cat-D01004174288P0210/?benchmark=1
Currently, it takes 0.2110 seconds to generate the page with PHP on the first page load. But out of this 0.21 seconds, 0.1865 is due to calling an external geolocation API. This is just because Cloudflare is currently disabled for our testing phase. When Cloudflare is enabled, we get the geolocation from $_SERVER["HTTP_CF_IPCOUNTRY"] so we avoid calling the external API and page load time goes down to around 0.01s
TTFB is not the only problem, we also have high LCP and it throws warnings in Google Search Console: https://prnt.sc/2PjB74ekUgdR
Our search ranking dropped a lot and we lost 50% of our search traffic since June 2021 and we suspect it could be partly due to that
How can I confirm whether the disk is I/O bound or not? I followed Plesk guide and here are the results:
https://support.plesk.com/hc/en-us/articles/360006437794-Plesk-for-Linux-server-is-running-slow-too-much-IO-interactivity
`[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=dir ect
5000+0 records in
5000+0 records out
2560000 bytes (2.6 MB, 2.4 MiB) copied, 13.634 s, 188 kB/s
[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=direct
5000+0 records in
5000+0 records out
2560000 bytes (2.6 MB, 2.4 MiB) copied, 14.1703 s, 181 kB/s
[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=512 count=5000 oflag=direct
5000+0 records in
5000+0 records out
2560000 bytes (2.6 MB, 2.4 MiB) copied, 14.0026 s, 183 kB/s
`
This seems pretty slow. Could it be because we run pagespeed from too many domains?
Latency seems fine:
```
[root@ns508279 ~]# dd if=/dev/zero of=/root/testfile bs=1024M count=1 oflag=direct
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.05548 s, 522 MB/s
```
however iotop looks perfectly fine. Monitored it for around 10 minutes and IO never went above 1%
https://prnt.sc/F65mP1MkTeBC
also no high %WA with top:
https://prnt.sc/Oe1hhgcqrNDC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1096614478
The shared-memory metadata cache does not get populated with images. It
just gets populated with metadata mapping the original image name and
context into which it is to be rendered into an optimized resource name.
The images are stored in memcached in your configuration. You can see the
hierarchy pictured here:
https://www.keinegotterkeinemeister.de/pagespeed_admin/cache#cache_struct
It look like you've got Prioritize Critical CSS enabled. I haven't
thought about this space for quite some time but I don't think we have much
else in PageSpeed I'd recommend. I"d try disabling defer_javascript -- it's
possible it's doing more harm than good to that metric.
Good luck!
On Mon, Apr 11, 2022 at 11:52 PM Ungov77 ***@***.***> wrote:
> There could be an IO issue because according to the Plesk guide, the first
> test I ran should get a speed of 142 MB/s but I got only 183 *kB*/s
>
> Memcached is now installed but I'm not sure how to check if it's correctly
> configured
> https://www.keinegotterkeinemeister.de/phpinfo.php
> I enabled statistics and I can see some data related to Memcached so it
> seems to be working:
> https://www.keinegotterkeinemeister.de/mod_pagespeed_statistics?memcached
>
> ModPagespeedMemcachedServers localhost:11211
> ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 51200
>
> Is 51200 enough? We have plenty of memory on the server and the websites
> are big with thousands of images
>
> It seems to have lowered TTFB, great! Now it is almost the same as with
> Pagespeed disabled. Thanks!
>
> Anything suggestions to improve LCP?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095951580>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAO2IPIHW2BALPOWN7TII53VETXQJANCNFSM5TEGVEXQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1097249240
I tried CTRL+F5 many many times but I didn't see any webp images in the code. As soon as I turn off Memcached, I see webp images after 2 refreshes. We let Memcached be enabled for 18 hours and didn't see any webp images on the homepage.
Memcached disabled, many webp images in the code:
view-source:https://www.keinegotterkeinemeister.de/
Memcached enabled for 18 hours with zero webp in the code:
view-source:https://www.ni-dieu-ni-maitre.com/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1098556400
Hi,
Makes sense. I didn't understand how it was faster with Memcached failing lol
I don't think the rewrite deadline is being overwritten. I have not modified any conf file, the entire config is in Plesk GUI except for ModPagespeedCreateSharedMemoryMetadataCache
I have looked at the histograms but I'm not sure how to interpret the data
You can check the admin section, I put it public temporarily again
https://www.keinegotterkeinemeister.de/pagespeed_admin/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1096667259
One more thing -- you probably should control access to /pagespeed_admin and /mod_pagespeed_statistics with allow/deny directives per https://www.modpagespeed.com/doc/admin#handlers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095856844
I was suspicious there may be spinning-disk latency but as you have an SSD I'm not sure. Nevertheless it might be worth trying redis/memcached, as we never really tuned the file-cache for SSD.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1099985385
hi @Ungov77
Take account that your filesystem cache was near of 40GB, if I remember +30 (Yes, I´m Longinos in the goolge group email list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1097233197
My first suspicion is that they just haven't been (re)converted into webp *yet*. By switching to memcached you effectively are flushing your cache. As you may have noticed, PageSpeed optimizes resources in the background when it learns they need to be optimized, but may serve the first few requests unoptimized. Is it possible that if you refresh your page a few times the images will start coming in as webp?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Lofesa commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Lofesa commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1099401496
Hi @Ungov77 @jmarantz
Take a look on this:
https://www.keinegotterkeinemeister.de/pagespeed_admin/cache#physical_cache
See evictions and current objects stored:
curr_items: 53457
evictions: 25654735
I think is a big amount of evistions.
You are using only a 64Mb memcache database
bytes: 63936177
In the config file for memcached (normaly /etc7memcached.conf or /etc/system/memcached.conf) you can increase the size of RAM memcached uses:
change `m 64` whit the size you will use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095951580
There could be an IO issue because according to the Plesk guide, the first test I ran should get a speed of 142 MB/s but I got only 183 **kB**/s
Memcached is now installed but I'm not sure how to check if it's correctly configured
https://www.keinegotterkeinemeister.de/phpinfo.php
I enabled statistics and I can see some data related to Memcached so it seems to be working:
https://www.keinegotterkeinemeister.de/mod_pagespeed_statistics?memcached
```
ModPagespeedMemcachedServers localhost:11211
ModPagespeedCreateSharedMemoryMetadataCache "/var/cache/mod_pagespeed/" 51200
```
Is 51200 enough? We have plenty of memory on the server and the websites are big with thousands of images
It seems to have lowered TTFB, great! Now it is almost the same as with Pagespeed disabled. Thanks!
Anything suggestions to improve LCP?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] jmarantz commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
jmarantz commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1095540226
Thanks for the data including the config; it's very helpful.
It does look like the performance is pretty good even from the US, despite the TTFB. I assume the server is in Germany with no US CDN.
Of course TTFB will never get faster with PageSpeed, only slower. The question is whether the optimizations it makes during that time provide a better user experience.
I visited view-source:https://www.keinegotterkeinemeister.de/tshirt-sabotage-black-cat-D01004174288P0210/?PageSpeedFilters=+debug to see what timing info PageSpeed injected at the end, and found this:
```
#NumFlushes 0
#EndDocument after 5906us
#Total Parse duration 5685us
#Total Render duration 411563us
#Total Idle duration 221us
No critical images detected.
The following filters were disabled for this request:
CriticalSelectorFilter: No critical selector info in cache
DelayImages
```
So it looks like your server spent 5.7 seconds in PageSpeed parsing your HTML -- that's mostly CPU bound. Then it spent 452ms rendering it -- that may be a combination of image optimization, network operations, waiting for asynchronous optimizations and URL fetches (depending on the cache setup) That leaves 1100-270= 378ms unaccounted for.
I'm guessing that most of this comes from the file cache. It's possible your disk is I/O bound. My main suggestion is to try deploying memcached or redis as an asynchronous cache. That should be faster than the file-cache.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-pagespeed-mod] Ungov77 commented on issue #2089: High TTFB with PageSpeed
Posted by GitBox <gi...@apache.org>.
Ungov77 commented on issue #2089:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2089#issuecomment-1099694906
Hi,
The config file aas in /etc/sysconfig/memcached
What size should we increase it to? We have 64gb of ram on this server
I increased it from 64 to 6400 for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pagespeed.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org