You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pagespeed.apache.org by GitBox <gi...@apache.org> on 2020/09/18 10:10:51 UTC
[GitHub] [incubator-pagespeed-mod] sourabh8Webscale opened a new issue #2039: ModPagespeedShardDomain not working for a non-optimised sharded request.
sourabh8Webscale opened a new issue #2039:
URL: https://github.com/apache/incubator-pagespeed-mod/issues/2039
**Whats is the Issue?**
Requests having sharded hostname and non-optimized resources(non-pagespeed resource) are not being mapped to origin domain.
Here is a glimpse of my configuration, I have provided a detailed configuration in the last section.
```
# CDN Sharding is enabled.
ModPagespeedShardDomain http://www.example.com http://cdn1-cname.example.com/ABC,http://cdn2-cname.example.com/ABC,http://cdn3-cname.example.com/ABC
ModPagespeedShardDomain http://www.alias-example.com http://cdn1-cname.example.com/DEF,http://cdn2-cname.example.com/DEF,http://cdn3-cname.example.com/DEF
# Map origin domain.
ModPagespeedMapOriginDomain 127.0.0.1 http://www.example.com
ModPagespeedMapOriginDomain 127.0.0.1 http://www.alias-example.com
```
`ABC`, `DEF` are our checksum to map the shard request to a correct hostname or alias.
**Problem statement:**
Requests like `http://www.cdn1-cname.example.com/ABC/ex/fonts/myfont.tiff` ends up at localhost with path as `/ABC/ex/fonts/myfonts.tiff` instead of `/ex/fonts/myfonts.tiff`.
**What is the expected output?**
I expect the URL `http://www.cdn1-cname.example.com/ABC/ex/fonts/myfont.tiff` to be mapped to its master domain
something like `http://www.example.com/ex/fonts/myfont.tiff` and then should request to its origin with URL something like
`http://localhost/ex/fonts/myfont.tiff`.
**What do you see instead?**
_For non-optimized and sharded resource:_
What I can observe is that when a request like `http://www.cdn1-cname.example.com/ABC/ex/fonts/myfont.tiff` reaches
the Instaweb_handler, it first checks if the requests is a pagespeed original URL or if such request has already been made
or if requests are a part of another request and it didn't find it to be anyone of them
then it creates a URL using `ap_construct_url`, which is similar to the current URL. Then it registers this
requests as original pagespeed request by registering it using`apr_table_setn`.
Then it checks is this URL is a pagespeed beacon request or if this URL is a pagespeed resource, which turns out to be false.
And this marks the URL to be a non-pagespeed URL using `apr_table_set`.
Pagespeed doesn't alter this URL and then `ProxyFixupHost` changes the host to `www.example.com` and updates the request to `http://www.example.com/ABC/ex/fonts/myfont.tiff` and `ProxyPass` directs it to balancer cluster which updates requests to http://localhost/ABC/ex/fonts/myfont.tiff`.
_For optimized resource:_
For example a sharded and optimized request like this
`http://www.cdn1-cname.example.com/ABC/ex/400x300a1.jpg.pagespeed.ic.hash.webp`
gets properly mapped to its master domain `http://www.example.com/ex/400x300a1.jpg.pagespeed.ic.hash.webp`
and then it gets searched in pagespeed cache if found then it returns the resource if not then
it tries to load the resource asynchronously by updating the URL to `http://www.cdn1-cname.example.com/ABC/ex/1.jpg` and then fetches the resource from the localhost with request `http://127.0.0.1/ex/1.jpg`.
**On what operating system?**
Ubuntu 16.04.7
**Which version of Apache?**
2.4.43
**Which version of pagespeed?**
v1.13.35.2
**What steps will reproduce the problem?**
Request a page in which there should resources which pagespeed should decide not to optimize , such as font files or small images the file formats which are failing for me are .tiff, .woff, .woff2, .png, .svg. and a few others.
This issue is coming in production where the customer site has a lot of resources but only above mentioned resources
are not being handled properly.
**My configuration:**
```
<VirtualHost 127.0.0.1:80 10.12x.x.3x:80>
ServerName www.example.com
ServerAlias www.alias-example.com
ConcurrentLimit xx
SuspendLimit 4xx
MaximumQueueTime 6x
DocumentRoot lg_vhosts/cname.example.com.80/www
SetEnvIfExpr true proxy-request-timeout=30
AllowEncodedSlashes NoDecode
ModPagespeedFileCachePath /var/cache/pagespeed/cname.example.com
ModPagespeedFileCacheSizeKb 10xxxx
WSGeoLookUpEnable 1
RemoteIPInternalProxy 127.0.0.1
LegacyTrust off
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/javascript text/javascript text/x-js text/x-javascript
<IfModule pagespeed_module>
ModPagespeed On
ModPagespeedEnableFilters insert_dns_prefetch
ModPagespeedEnableFilters add_instrumentation
ModPagespeedUrlValuedAttribute img data-src image
ModPagespeedFetchHttps enable
ModPagespeedDomain www.example.com
ModPagespeedDomain www.alias-example.com
ModPagespeedDomain https://www.example.com
ModPagespeedDomain https://www.alias-example.com
# CDN Sharding is enabled.
ModPagespeedShardDomain http://www.example.com http://cdn1-cname.example.com/ABC,http://cdn2-cname.example.com/ABC,http://cdn3-cname.example.com/ABC
ModPagespeedShardDomain http://www.alias-example.com http://cdn1-cname.example.com/DEF,http://cdn2-cname.example.com/DEF,http://cdn3-cname.example.com/DEF
ModPagespeedShardDomain https://www.example.com https://cdn1-cname.example.com/ABC,https://cdn2-cname.example.com/ABC,https://www.cdn3-cname.example.com/ABC
ModPagespeedShardDomain https://www.alias-example.com https://cdn1-cname.example.com/DEF,https://cdn2-cname.example.com/DEF,https://cdn3-cname.example.com/DEF
# Map origin domain.
ModPagespeedMapOriginDomain 127.0.0.1 http://www.example.com
ModPagespeedMapOriginDomain 127.0.0.1 http://www.alias-example.com
</IfModule>
ProxyPreserveHost On
ProxySourceAddress 10.12x.0.xx
SetEnvIfExpr "%{CONN_REMOTE_ADDR} = '127.0.0.1'" !proxy-initial-not-pooled
<Proxy balancer://cluster_0_0>
BalancerMember http://10x.1xx.1xx.1xx:80 route=0 timeout=600 retry=0 connectiontimeout=60 loadfactor=1
ProxySet lbmethod=bybusyness
</Proxy>
# Set to default balancer if not already set
SetEnvIfExpr "-z reqenv('LB')" LB=cluster_0_0
ProxyPassInterpolateEnv On
SSLProxyEngine On
RequestHeader set X-Forwarded-Proto http
Protocols http/1.1
</VirtualHost>
<VirtualHost 127.0.0.1:80 10.12x.x.3x:80>
ServerName cname.example.com
ConcurrentLimit 40
SuspendLimit 400
MaximumQueueTime 60
Session off
DocumentRoot lg_vhosts/cname.example.com.80/www
AllowEncodedSlashes NoDecode
ServerAlias cdn1-cname.example.com
ServerAlias cdn2-cname.example.com
ServerAlias cdn3-cname.example.com
SecWebAppId xyz
ModPagespeedFileCachePath /var/cache/pagespeed/cname.example.com
ModPagespeedFileCacheSizeKb 10xxxxx
WSGeoLookUpEnable 1
RemoteIPInternalProxy 127.0.0.1
LegacyTrust off
AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/javascript text/javascript text/x-js text/x-javascript
<IfModule pagespeed_module>
ModPagespeed On
ModPagespeedEnableFilters insert_dns_prefetch
ModPagespeedEnableFilters add_instrumentation
ModPagespeedUrlValuedAttribute img data-src image
ModPagespeedFetchHttps enable
ModPagespeedDomain www.example.com
ModPagespeedDomain www.alias-example.com
ModPagespeedDomain https:// www.example.com
ModPagespeedDomain https://www.alias-example.com
# CDN Sharding is enabled.
ModPagespeedShardDomain http://www.example.com http://cdn1-cname.example.com/ABC,http://cdn2-cname.example.com/ABC,http://cdn3-cname.example.com/ABC
ModPagespeedShardDomain http://www.alias-example.com http://cdn1-cname.example.com/DEF,http://cdn2-cname.example.com/DEF,http://cdn3-cname.example.com/DEF
# Optimize HTTPS is enabled as well, so we need to define https shards.
ModPagespeedShardDomain https://www.example.com https://www.cdn1-cname.example.com/ABC,https://cdn2-cname.example.com/ABC,https://cdn3-cname.example.com/ABC
ModPagespeedShardDomain https://www.alias-example.com https://www.cdn1-cname.example.com/DEF,https://cdn2-cname.example.com/DEF,https://cdn3-cname.example.com/DEF
# Map origin domain.
ModPagespeedMapOriginDomain 127.0.0.1 http://www.example.com
ModPagespeedMapOriginDomain 127.0.0.1 http://www.alias-example.com
</IfModule>
ProxyFixupHost www.example.com
SetEnvIfExpr true !proxy-initial-not-pooled
<Proxy balancer://cluster_0_0>
BalancerMember http://127.0.0.1:80
ProxySet lbmethod=bybusyness
</Proxy>
ProxyPass / balancer://cluster_0_0/
RequestHeader set X-Forwarded-Proto http
Protocols http/1.1
</VirtualHost>
```
We have created two virtual hosts one for handling CDN requests and one for the actual server. The CDN requests come first and then it gets resolved to the actual hostname and then request is send to the load balancer which picks the server based on busyness.
**Things I have tried:**
1. I have experimented with the configuration with the following changes
added
` ModPagespeedShardDomain 127.0.0.1 www.cdn1-cname.example.com/ABC www.example.com` to vhost file, but it gave me similar results.
2. I tried some code changes in `Instawebhandler` to map the sharded requests to its correct hostname, when its a sharded and non-pagespeed requests, but it failed to change the request and the request kept reverting to the original request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org