[jira] [Comment Edited] (TAP5-2201) Serious issue with assets and checksums - different for same file

["Magnus Kvalheim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TAP5-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796629#comment-13796629 ] 

Magnus Kvalheim edited comment on TAP5-2201 at 10/16/13 10:30 AM:
------------------------------------------------------------------

Okay, so believe I've found the root cause of this now.

I think it's best explained by describing the 5.3.x release procedures.
=5.3.x=
We have a cluster of application server and the release of new versions is not atomic. So there is a transition where some run old version and others new.
Tapestry assets are normally served through cdn cloudfront with tapestry as origin on a cookieless domain. That is - cloudfront hit's the loadbalancer.
So in order to prevent mixing of assets between versions while upgrading, we temporarily disable the cdn and serve them though same domain with sticky sessions.
Once all servers are upgraded we can re-enable cdn and all assets are available from any node - so no need for sticky sessions anymore.

To make this work we have an AssetPathConverter which is allowing us to make asset domain changes in running system - without reboot. It caches for 10min, but then checks if the assets path/domain has changed.

=5.4=
What we see with 5.4 is that this approach is no longer working. The main problem seems to be with the behavior of expanding css paths to full path.
This gives us a sort of chicken and egg problem. It will serve fine as long as cdn is disabled and sticky sessions are used, but when it's re-enabled things start failing.
For one - the css is cached and css expanded paths are not updated. Secondly, the assetpathconverter does not change domain at the same time across cluster (has 10 minute cache). 
At this point I'm not sure how we can reliably make a cluster upgrade, which is not atomic, and still take care of the cdn assets.

I think it might work though if the css attributes were not expanded to full url, but absolute to the host/domain of it's css.
That said, I realize now that it's probably our own CDNAssetPathConverter that is invoked to resolve the css url attributes. 
A possible solution then could be to disable it for those cases - if there is some way of identifying that it's parent source is a css or already on the correct domain...

I do like the CloudFront approach, and think it could be quite useful for others if they only knew about it and how it works.
It's easy to configure and there is no need to manually upload/put assets on a cdn as they are retrieved lazily by configuring app (loadbalancer) as origin.
Just need to work out how it can be achieved in 5.4...
Got any ideas?




was (Author: magnus@kvalheim.dk):
Okay, so believe I've found the root cause of this now.

I think it's best explained by describing the 5.3.x release procedures.
=5.3.x=
We have a cluster of application server and the release of new versions is not atomic. So there is a transition where some run old version and others new.
Tapestry assets are normally served through cdn cloudfront with tapestry as origin on a cookieless domain. That is - cloudfront hit's the loadbalancer.
So in order to prevent mixing of assets between versions while upgrading, we temporarily disable the cdn and serve them though same domain with sticky sessions.
Once all servers are upgraded we can re-enable cdn and all assets are available from any node - so no need for sticky sessions anymore.

To make this work we have an AssetPathConverter which is allowing us to make asset domain changes in running system - without reboot. It caches for 10min, but then checks if the cdn has changed.

=5.4=
What we see with 5.4 is that this approach is no longer working. The main problem seems to be with the behavior of expanding css paths to full path.
This gives us a sort of chicken and egg problem. It will serve fine as long as cdn is disabled and sticky sessions are used, but when it's re-enabled things start failing.
For one - the css is cached and css expanded paths are not updated. Secondly, the assetpathconverter does not change domain at the same time across cluster (has 10 minute cache). 
At this point I'm not sure how we can reliably make a cluster upgrade, which is not atomic, and still take care of the cdn assets.

I think it might work though if the css attributes were not expanded to full url, but absolute to the host/domain of it's css.
That said, I realize now that it's probably our own CDNAssetPathConverter that is invoked to resolve the css url attributes. 
A possible solution then could be to disable it for those cases - if there is some way of identifying that it's parent source is a css or already on the correct domain...

I do like the CloudFront approach, and think it could be quite useful for others if they only knew about it and how it works.
It's easy to configure and there is no need to manually upload/put assets on a cdn as they are retrieved lazily by configuring app (loadbalancer) as origin.
Just need to work out how it can be achieved in 5.4...
Got any ideas?



> Serious issue with assets and checksums - different for same file
> -----------------------------------------------------------------
>
>                 Key: TAP5-2201
>                 URL: https://issues.apache.org/jira/browse/TAP5-2201
>             Project: Tapestry 5
>          Issue Type: Bug
>          Components: tapestry-core
>    Affects Versions: 5.4
>            Reporter: Magnus Kvalheim
>              Labels: 5.4.22
>         Attachments: bootstrap.server2.css, bootstrap.server3.css, server2.png, server3.png
>
>
> Hi everybody.
> Today we've launched a website based on 5.4. We're very exited about the upcoming release(5.4) and I'll post separately about our experiences (mostly great).
> Post release we've identified a potential serious issue related to assets and their checksums.
> What we see is that a handful of the assets generate different hashes for the same file.
> =Example: bootstrap.css=
> Server 1: 
> /asset.gz/meta/92ffb14a/tapestry5/bootstrap_3_0_0/css/bootstrap.css
> Server 2:
> /asset.gz/meta/5787e482/tapestry5/bootstrap_3_0_0/css/bootstrap.css
> Server 3:
> /asset.gz/meta/f5e7c535/tapestry5/bootstrap_3_0_0/css/bootstrap.css
> Server 3 - restart:
> /asset.gz/meta/219ee41e/tapestry5/bootstrap_3_0_0/css/bootstrap.css
> We also see the same behaviour for the non gzip version of bootstrap.css.
> It is not only for /meta/
> =JCarouselWrapper.css=
> /asset/app/f59da774/mixins/ui/JCarouselWrapper.css
> /asset/app/6ddc92ee/mixins/ui/JCarouselWrapper.css
> As you can see - we're load balanced with app served from several nodes.
> Normally I'd serve these through CloudFront on a cookieless domain (with tapestry as origin), but it's not possible as load balanced assets could hit 'wrong' server and get the 404 instead.
> So for now they are served through website domain with sticky sessions - and pray that it don't cause us problems... :)
> All are served with same web container: 
> Apache Tomcat/7.0.39
> JDK 1.7.0_11



--
This message was sent by Atlassian JIRA
(v6.1#6144)