Re: Caching / Ehcache

[Marc-Aurèle Brothier <m...@brothier.org>]

I wasn't in Montreal so I wasn't aware of the willing to change the JPA layer. That's definitively a good thing.

> On 30 Oct 2018, at 17:05, Rafael Weingärtner <ra...@gmail.com> wrote:
> 
> Hello Marc,
> 
> It is said to hear that you might not be active as you used to be. However,
> I do wish you success in your new path.
> 
> Now, regarding the ehCache. +1 to remove it. Were you in Montreal? This
> year in Montreal, one of the things we discussed is to move away from this
> ad-hoc JPA solution that we have to a standardized and proved one (e.g.
> Spring data + some commonly used JAP implementation such as Eclipse link,
> OpenJPA, and others). AS a first step towards that, we could disable and
> remove the use of ehCache.
> 
> 
> 
> P.S. sorry the late reply, but I have been busy with others topics not
> related to ACS.
> 
> On Sun, Oct 28, 2018 at 2:17 PM Daan Hoogland <da...@gmail.com>
> wrote:
> 
>> Sorry to hear you are reducing your activity on CloudStack, Marc-Aurele.
>> Hope you fare well.
>> 
>> On Sat, Oct 27, 2018 at 10:10 PM Marc-Aurèle Brothier <m...@brothier.org>
>> wrote:
>> 
>>> Hi everyone,
>>> 
>>> (Again as the email formatting has been removed and was hard to read -
>>> I hope it will be better this time).
>>> 
>>> While trying to lower the DB load for CloudStack I did some long
>>> testing and here are my outcomes for the current cache mechanism in
>>> CloudStack.
>>> 
>>> I would be interested to hear from people who try to customize the
>>> ehcache configuration in CS. A PR (
>>> https://github.com/apache/cloudstack/pull/2913) is also open to
>>> desactivate (before deleting) ehcache in CS, read below to understand
>>> why.
>>> 
>>> 
>>> # Problems
>>> 
>>> The code in CS does not seem to fit any caching mechanism especially
>>> due to the homemade DAO code. The main 3 flaws are the following:
>>> 
>>> ## Entities are not expected to be shared
>>> 
>>> There is quite a lot of code with method calls passing entity IDs value
>>> as long, which does some object fetching. Without caching, this
>>> behavior will create distinct objects each time an entity with the same
>>> ID is fetched. With the cache enabled, the same object will be shared
>>> among those methods. It has been seen that it does generate some side
>>> effects where code still expected unchanged entity attributes after
>>> calling different methods thus generating exception/bugs.
>>> 
>>> ## DAO update operations are using search queries
>>> 
>>> Some part of the code are updating entities based on a search query,
>>> therefore the whole cache must be invalidated (see GenericDaoBase:
>>> public int update(UpdateBuilder ub, final SearchCriteria<?> sc, Integer
>>> rows);).
>>> 
>>> ## Entities based on views joining multiple tables
>>> 
>>> There are quite a lot of entities based on SQL views joining multiple
>>> entities in a same object. Enabling caching on those would require a
>>> mechanism to link and cross-remove related objects whenever one of the
>>> sub-entity is changed.
>>> 
>>> 
>>> # Final word
>>> 
>>> Based on the previously discussed points, the best approach IMHO would
>>> be to move out of the custom DAO framework in CS and use a well known
>>> one. It will handle caching well and the joins made by the views in the
>>> code. It's not an easy change, but it will fix along a lot of issues
>>> and add a proven / robust framework to an important part of the code.
>>> 
>>> The work to change the DAO layer is a huge task, I don't know how / who
>>> will perform it.
>>> 
>>> What are the proposals for a new DAO framework ?
>>> 
>>> 
>>> FYI I will stop working for Exoscale at the end of the month, so I
>>> won't be able to tackle such challenge as I won't be working with CS
>>> anymore. I'll try my best to continue looking at the project to give my
>>> insights and share the experienced I have with CS.
>>> 
>>> 
>>> Marc-Aurèle
>>> 
>>> 
>> 
>> --
>> Daan
>> 
> 
> 
> --
> Rafael Weingärtner