You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@polygene.apache.org by zhuangmz08 <zh...@qq.com> on 2016/04/28 07:21:42 UTC
cache mechanism?
Hi,
I would like to assembling the following layers (from up to bottm) [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store / index query)] [Config Layer].
I can assemble this layers correctly. I can use (domainModule.newUnitOfWork) to write and query data. I can find the entities stored in Postgres.
However, I'm confused about caching mechanism here?
When did the system put entity into cache? When newEntityInstance and uow.complete? When newQurey.find and uow.complete?
When did the system update the cache? When entity.property.set and uow.complete?
Thanks a lot.
Re: cache mechanism?
Posted by Paul Merlin <pa...@nosphere.org>.
First cut of the SQL EntityStore in 1.x used to use JSONMapEntityStoreMixin.
Stan and I changed that in 2.x in an attempt to have SQL Index/Query
collaborate closely with SQL EntityStore. IIRC one of the idea was to
allow the former to reuse primary keys generated by the latter, sharing
them via EntityState and other performance optimisations.
About having a way to have a SQL EntityStore and Index/Query working
closely, it may be interesting for SQL only deployments but it should
not be at the cost of the nice features Zest brings to
extensions.Optimizing the full-SQL scenario should be an opt-in feature.
I think we should have a *simple* SQL EntityStore, based on
JSONMapEntityStoreMixin so it can directly leverage all the associated
features like caching.See ZEST-153
Stanislav Muhametsin a �crit :
> I think Paul was more involved with SQL EntityStore (while I was
> crunching out the SQL Indexing), so he might help with this.
> IIRC, JSONMapEntityStoreMixin at least back then (if it was even
> called like that) had some issues which made it unusable with SQL ES.
>
> On 03/05/2016 16:50, Niclas Hedhman wrote:
>> I just checked, and yes you are right, the SQL EntityStore isn't using
>> JSONMapEntityStoreMixin nor does it use the Cache SPI.
>>
>> I suggest that you add a Jira issue (if it doesn't already exist) and if
>> you are a bit daring perhaps help out fixing it ;-)
>>
>> Cheers
>> Niclas
>>
>> On Tue, May 3, 2016 at 3:33 PM, zhuangmz08 <zh...@qq.com> wrote:
>>
>>> Thanks to Kent, I deleted the Cache Layer and build a cache module
>>> in the
>>> Persist Layer. I found that SQL entities doesn't support caching. Only
>>> JSONMapEntityStoreMixin support caching?
>>>
>>>
>>>
>>>
>>> ------------------ \u539f\u59cb\u90ae\u4ef6 ------------------
>>> \u53d1\u4ef6\u4eba: "Niclas Hedhman";<he...@gmail.com>;
>>> \u53d1\u9001\u65f6\u95f4: 2016\u5e744\u670828\u65e5(\u661f\u671f\u56db) \u665a\u4e0a9:41
>>> \u6536\u4ef6\u4eba: "dev"<de...@zest.apache.org>;
>>>
>>> \u4e3b\u9898: Re: cache mechanism?
>>>
>>>
>>>
>>> Adding to Kent's good answer;
>>>
>>> 1. I think Kebt wasn't explicit enough; You shouldn't have a Caching
>>> Layer.
>>> Caching is part of the persistence and should be in the same layer.
>>>
>>> 2. Indexing isn't synchronous with persistence, it is what is called
>>> Eventually Consistent. You even have a lot of custom control over
>>> how it
>>> should work. But it means that if you run a query within a
>>> UnitOfWork the
>>> indexing has not yet seen the "current" changes, and even if you run it
>>> directly after .complete(), you might not get the latest changes
>>> yet. Mind
>>> you, I have some thoughts about making some changes in how all of this
>>> works, so ot might change in 3.0 to some degree (or not).
>>>
>>> Niclas
>>>
>>>
>>> On Apr 28, 2016 14:43, "Kent S�lvsten" <ke...@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> The cache is wrapping the EntityStore (in your case the Postgres
>>>> access)
>>>> - if you happen to be familiar to eg. Hibernate, it is similar to a
>>>> Hibernate level 2 cache.
>>>>
>>>> This means:
>>>>
>>>> If you create a new entity, the entity will be created at the
>>>> cache at
>>>> uow.complete() - immediately after writing to postgres.
>>>> If you load an existing entity (by id) the cache is checked before
>>>> reading from Postgres.
>>>> Updating that entity is similar to creation: the cache will be updated
>>>> at uow.complete() - nothing happens when the entity becomes "dirty".
>>>>
>>>> Querying is a bit more complex:
>>>> When you query for entities, under the hood the query will be
>>>> performed
>>>> against an indexer returning references (id's) for entities.
>>>> These references are converted to full entities using either the cache
>>>> or postgres.
>>>>
>>>> So the short story: The cache should contain the same state as your
>>>> Postgres entitystore.
>>>>
>>>> Another thing is that in you configuration, the EntityStore should
>>>> have
>>>> access to the cache - otherwise the cache will simply be silently
>>>> ignored by the persistence mechanism.
>>>>
>>>> /Kent
>>>>
>>>>
>>>> Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
>>>>> Hi,
>>>>>
>>>>>
>>>>> I would like to assembling the following layers (from up to bottm)
>>>> [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store /
>>>> index query)] [Config Layer].
>>>>> I can assemble this layers correctly. I can use
>>>> (domainModule.newUnitOfWork) to write and query data. I can find the
>>>> entities stored in Postgres.
>>>>> However, I'm confused about caching mechanism here?
>>>>> When did the system put entity into cache? When newEntityInstance and
>>>> uow.complete? When newQurey.find and uow.complete?
>>>>> When did the system update the cache? When entity.property.set and
>>>> uow.complete?
>>>>>
>>>>> Thanks a lot.
>>>>
>>
>>
>
Re: cache mechanism?
Posted by Stanislav Muhametsin <st...@zest.mail.kapsi.fi>.
I think Paul was more involved with SQL EntityStore (while I was
crunching out the SQL Indexing), so he might help with this.
IIRC, JSONMapEntityStoreMixin at least back then (if it was even called
like that) had some issues which made it unusable with SQL ES.
On 03/05/2016 16:50, Niclas Hedhman wrote:
> I just checked, and yes you are right, the SQL EntityStore isn't using
> JSONMapEntityStoreMixin nor does it use the Cache SPI.
>
> I suggest that you add a Jira issue (if it doesn't already exist) and if
> you are a bit daring perhaps help out fixing it ;-)
>
> Cheers
> Niclas
>
> On Tue, May 3, 2016 at 3:33 PM, zhuangmz08 <zh...@qq.com> wrote:
>
>> Thanks to Kent, I deleted the Cache Layer and build a cache module in the
>> Persist Layer. I found that SQL entities doesn't support caching. Only
>> JSONMapEntityStoreMixin support caching?
>>
>>
>>
>>
>> ------------------ \u539f\u59cb\u90ae\u4ef6 ------------------
>> \u53d1\u4ef6\u4eba: "Niclas Hedhman";<he...@gmail.com>;
>> \u53d1\u9001\u65f6\u95f4: 2016\u5e744\u670828\u65e5(\u661f\u671f\u56db) \u665a\u4e0a9:41
>> \u6536\u4ef6\u4eba: "dev"<de...@zest.apache.org>;
>>
>> \u4e3b\u9898: Re: cache mechanism?
>>
>>
>>
>> Adding to Kent's good answer;
>>
>> 1. I think Kebt wasn't explicit enough; You shouldn't have a Caching Layer.
>> Caching is part of the persistence and should be in the same layer.
>>
>> 2. Indexing isn't synchronous with persistence, it is what is called
>> Eventually Consistent. You even have a lot of custom control over how it
>> should work. But it means that if you run a query within a UnitOfWork the
>> indexing has not yet seen the "current" changes, and even if you run it
>> directly after .complete(), you might not get the latest changes yet. Mind
>> you, I have some thoughts about making some changes in how all of this
>> works, so ot might change in 3.0 to some degree (or not).
>>
>> Niclas
>>
>>
>> On Apr 28, 2016 14:43, "Kent S�lvsten" <ke...@gmail.com> wrote:
>>
>>> Hi
>>>
>>> The cache is wrapping the EntityStore (in your case the Postgres access)
>>> - if you happen to be familiar to eg. Hibernate, it is similar to a
>>> Hibernate level 2 cache.
>>>
>>> This means:
>>>
>>> If you create a new entity, the entity will be created at the cache at
>>> uow.complete() - immediately after writing to postgres.
>>> If you load an existing entity (by id) the cache is checked before
>>> reading from Postgres.
>>> Updating that entity is similar to creation: the cache will be updated
>>> at uow.complete() - nothing happens when the entity becomes "dirty".
>>>
>>> Querying is a bit more complex:
>>> When you query for entities, under the hood the query will be performed
>>> against an indexer returning references (id's) for entities.
>>> These references are converted to full entities using either the cache
>>> or postgres.
>>>
>>> So the short story: The cache should contain the same state as your
>>> Postgres entitystore.
>>>
>>> Another thing is that in you configuration, the EntityStore should have
>>> access to the cache - otherwise the cache will simply be silently
>>> ignored by the persistence mechanism.
>>>
>>> /Kent
>>>
>>>
>>> Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
>>>> Hi,
>>>>
>>>>
>>>> I would like to assembling the following layers (from up to bottm)
>>> [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store /
>>> index query)] [Config Layer].
>>>> I can assemble this layers correctly. I can use
>>> (domainModule.newUnitOfWork) to write and query data. I can find the
>>> entities stored in Postgres.
>>>> However, I'm confused about caching mechanism here?
>>>> When did the system put entity into cache? When newEntityInstance and
>>> uow.complete? When newQurey.find and uow.complete?
>>>> When did the system update the cache? When entity.property.set and
>>> uow.complete?
>>>>
>>>> Thanks a lot.
>>>
>
>
Re: cache mechanism?
Posted by Niclas Hedhman <ni...@hedhman.org>.
I just checked, and yes you are right, the SQL EntityStore isn't using
JSONMapEntityStoreMixin nor does it use the Cache SPI.
I suggest that you add a Jira issue (if it doesn't already exist) and if
you are a bit daring perhaps help out fixing it ;-)
Cheers
Niclas
On Tue, May 3, 2016 at 3:33 PM, zhuangmz08 <zh...@qq.com> wrote:
> Thanks to Kent, I deleted the Cache Layer and build a cache module in the
> Persist Layer. I found that SQL entities doesn't support caching. Only
> JSONMapEntityStoreMixin support caching?
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Niclas Hedhman";<he...@gmail.com>;
> 发送时间: 2016年4月28日(星期四) 晚上9:41
> 收件人: "dev"<de...@zest.apache.org>;
>
> 主题: Re: cache mechanism?
>
>
>
> Adding to Kent's good answer;
>
> 1. I think Kebt wasn't explicit enough; You shouldn't have a Caching Layer.
> Caching is part of the persistence and should be in the same layer.
>
> 2. Indexing isn't synchronous with persistence, it is what is called
> Eventually Consistent. You even have a lot of custom control over how it
> should work. But it means that if you run a query within a UnitOfWork the
> indexing has not yet seen the "current" changes, and even if you run it
> directly after .complete(), you might not get the latest changes yet. Mind
> you, I have some thoughts about making some changes in how all of this
> works, so ot might change in 3.0 to some degree (or not).
>
> Niclas
>
>
> On Apr 28, 2016 14:43, "Kent Sølvsten" <ke...@gmail.com> wrote:
>
> > Hi
> >
> > The cache is wrapping the EntityStore (in your case the Postgres access)
> > - if you happen to be familiar to eg. Hibernate, it is similar to a
> > Hibernate level 2 cache.
> >
> > This means:
> >
> > If you create a new entity, the entity will be created at the cache at
> > uow.complete() - immediately after writing to postgres.
> > If you load an existing entity (by id) the cache is checked before
> > reading from Postgres.
> > Updating that entity is similar to creation: the cache will be updated
> > at uow.complete() - nothing happens when the entity becomes "dirty".
> >
> > Querying is a bit more complex:
> > When you query for entities, under the hood the query will be performed
> > against an indexer returning references (id's) for entities.
> > These references are converted to full entities using either the cache
> > or postgres.
> >
> > So the short story: The cache should contain the same state as your
> > Postgres entitystore.
> >
> > Another thing is that in you configuration, the EntityStore should have
> > access to the cache - otherwise the cache will simply be silently
> > ignored by the persistence mechanism.
> >
> > /Kent
> >
> >
> > Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
> > > Hi,
> > >
> > >
> > > I would like to assembling the following layers (from up to bottm)
> > [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store /
> > index query)] [Config Layer].
> > > I can assemble this layers correctly. I can use
> > (domainModule.newUnitOfWork) to write and query data. I can find the
> > entities stored in Postgres.
> > > However, I'm confused about caching mechanism here?
> > > When did the system put entity into cache? When newEntityInstance and
> > uow.complete? When newQurey.find and uow.complete?
> > > When did the system update the cache? When entity.property.set and
> > uow.complete?
> > >
> > >
> > > Thanks a lot.
> >
> >
>
--
Niclas Hedhman, Software Developer
http://zest.apache.org - New Energy for Java
回复： cache mechanism?
Posted by zhuangmz08 <zh...@qq.com>.
Thanks to Kent, I deleted the Cache Layer and build a cache module in the Persist Layer. I found that SQL entities doesn't support caching. Only JSONMapEntityStoreMixin support caching?
------------------ 原始邮件 ------------------
发件人: "Niclas Hedhman";<he...@gmail.com>;
发送时间: 2016年4月28日(星期四) 晚上9:41
收件人: "dev"<de...@zest.apache.org>;
主题: Re: cache mechanism?
Adding to Kent's good answer;
1. I think Kebt wasn't explicit enough; You shouldn't have a Caching Layer.
Caching is part of the persistence and should be in the same layer.
2. Indexing isn't synchronous with persistence, it is what is called
Eventually Consistent. You even have a lot of custom control over how it
should work. But it means that if you run a query within a UnitOfWork the
indexing has not yet seen the "current" changes, and even if you run it
directly after .complete(), you might not get the latest changes yet. Mind
you, I have some thoughts about making some changes in how all of this
works, so ot might change in 3.0 to some degree (or not).
Niclas
On Apr 28, 2016 14:43, "Kent Sølvsten" <ke...@gmail.com> wrote:
> Hi
>
> The cache is wrapping the EntityStore (in your case the Postgres access)
> - if you happen to be familiar to eg. Hibernate, it is similar to a
> Hibernate level 2 cache.
>
> This means:
>
> If you create a new entity, the entity will be created at the cache at
> uow.complete() - immediately after writing to postgres.
> If you load an existing entity (by id) the cache is checked before
> reading from Postgres.
> Updating that entity is similar to creation: the cache will be updated
> at uow.complete() - nothing happens when the entity becomes "dirty".
>
> Querying is a bit more complex:
> When you query for entities, under the hood the query will be performed
> against an indexer returning references (id's) for entities.
> These references are converted to full entities using either the cache
> or postgres.
>
> So the short story: The cache should contain the same state as your
> Postgres entitystore.
>
> Another thing is that in you configuration, the EntityStore should have
> access to the cache - otherwise the cache will simply be silently
> ignored by the persistence mechanism.
>
> /Kent
>
>
> Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
> > Hi,
> >
> >
> > I would like to assembling the following layers (from up to bottm)
> [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store /
> index query)] [Config Layer].
> > I can assemble this layers correctly. I can use
> (domainModule.newUnitOfWork) to write and query data. I can find the
> entities stored in Postgres.
> > However, I'm confused about caching mechanism here?
> > When did the system put entity into cache? When newEntityInstance and
> uow.complete? When newQurey.find and uow.complete?
> > When did the system update the cache? When entity.property.set and
> uow.complete?
> >
> >
> > Thanks a lot.
>
>
Re: cache mechanism?
Posted by Niclas Hedhman <he...@gmail.com>.
Adding to Kent's good answer;
1. I think Kebt wasn't explicit enough; You shouldn't have a Caching Layer.
Caching is part of the persistence and should be in the same layer.
2. Indexing isn't synchronous with persistence, it is what is called
Eventually Consistent. You even have a lot of custom control over how it
should work. But it means that if you run a query within a UnitOfWork the
indexing has not yet seen the "current" changes, and even if you run it
directly after .complete(), you might not get the latest changes yet. Mind
you, I have some thoughts about making some changes in how all of this
works, so ot might change in 3.0 to some degree (or not).
Niclas
On Apr 28, 2016 14:43, "Kent Sølvsten" <ke...@gmail.com> wrote:
> Hi
>
> The cache is wrapping the EntityStore (in your case the Postgres access)
> - if you happen to be familiar to eg. Hibernate, it is similar to a
> Hibernate level 2 cache.
>
> This means:
>
> If you create a new entity, the entity will be created at the cache at
> uow.complete() - immediately after writing to postgres.
> If you load an existing entity (by id) the cache is checked before
> reading from Postgres.
> Updating that entity is similar to creation: the cache will be updated
> at uow.complete() - nothing happens when the entity becomes "dirty".
>
> Querying is a bit more complex:
> When you query for entities, under the hood the query will be performed
> against an indexer returning references (id's) for entities.
> These references are converted to full entities using either the cache
> or postgres.
>
> So the short story: The cache should contain the same state as your
> Postgres entitystore.
>
> Another thing is that in you configuration, the EntityStore should have
> access to the cache - otherwise the cache will simply be silently
> ignored by the persistence mechanism.
>
> /Kent
>
>
> Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
> > Hi,
> >
> >
> > I would like to assembling the following layers (from up to bottm)
> [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store /
> index query)] [Config Layer].
> > I can assemble this layers correctly. I can use
> (domainModule.newUnitOfWork) to write and query data. I can find the
> entities stored in Postgres.
> > However, I'm confused about caching mechanism here?
> > When did the system put entity into cache? When newEntityInstance and
> uow.complete? When newQurey.find and uow.complete?
> > When did the system update the cache? When entity.property.set and
> uow.complete?
> >
> >
> > Thanks a lot.
>
>
Re: cache mechanism?
Posted by Kent Sølvsten <ke...@gmail.com>.
Hi
The cache is wrapping the EntityStore (in your case the Postgres access)
- if you happen to be familiar to eg. Hibernate, it is similar to a
Hibernate level 2 cache.
This means:
If you create a new entity, the entity will be created at the cache at
uow.complete() - immediately after writing to postgres.
If you load an existing entity (by id) the cache is checked before
reading from Postgres.
Updating that entity is similar to creation: the cache will be updated
at uow.complete() - nothing happens when the entity becomes "dirty".
Querying is a bit more complex:
When you query for entities, under the hood the query will be performed
against an indexer returning references (id's) for entities.
These references are converted to full entities using either the cache
or postgres.
So the short story: The cache should contain the same state as your
Postgres entitystore.
Another thing is that in you configuration, the EntityStore should have
access to the cache - otherwise the cache will simply be silently
ignored by the persistence mechanism.
/Kent
Den 28-04-2016 kl. 07:21 skrev zhuangmz08:
> Hi,
>
>
> I would like to assembling the following layers (from up to bottm) [Domain Layer] [Cache Layer (memcached)] [Persist Layer(entity store / index query)] [Config Layer].
> I can assemble this layers correctly. I can use (domainModule.newUnitOfWork) to write and query data. I can find the entities stored in Postgres.
> However, I'm confused about caching mechanism here?
> When did the system put entity into cache? When newEntityInstance and uow.complete? When newQurey.find and uow.complete?
> When did the system update the cache? When entity.property.set and uow.complete?
>
>
> Thanks a lot.