You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Raymond Wilson <ra...@trimble.com> on 2017/06/11 23:14:07 UTC
RE: Write behind using Grid Gain
Hi Pavel,
Thanks for the blog – it explains it quite well.
I have a slightly different use case where groups of records within a much
larger data set are clustered together for efficiency (ie: each of the
cached items in the Ignite grid cache has significant internal structure).
You can think of them as a large number of smallish files (a few Kb to a
few Mb), but file systems don’t like lots of small files.
I have a legacy implementation that houses these small files within a
single larger file, but wanted to know if there was a clean way of
supporting the same structure using the Ignite read/write through support,
perhaps with another system providing relatively transparent persistency
semantics but which does not use a DB to store the data.
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Saturday, May 27, 2017 5:03 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
I've decided to write a blog post, since this topic seems to be in demand:
https://ptupitsyn.github.io/Ado-Net-Cache-Store/
Code:
https://github.com/ptupitsyn/ignite-net-examples/tree/master/AdoNetCacheStore
Let me know if this helps!
On Fri, May 26, 2017 at 3:50 PM, Chetan D <cc...@gmail.com> wrote:
Thank you Pavel.
waiting for your response.
On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <pt...@apache.org>
wrote:
To give everyone context, this is not about GridGain, but about Apache
Ignite.
The blog post in question is
https://ptupitsyn.github.io/Entity-Framework-Cache-Store/
Chetan, I'll prepare an example with Ignite 2.0 / ado.net and post it some
time later.
Pavel
On Fri, May 26, 2017 at 2:32 PM, Chetan D <cc...@gmail.com> wrote:
++ User List
any help much appreciated.
Thanks And Regards
Chetan D
---------- Forwarded message ----------
From: *Pavel Tupitsyn* <pt...@apache.org>
Date: Fri, May 26, 2017 at 4:38 PM
Subject: Re: Write behind using Grid Gain
To: Chetan D <cc...@gmail.com>
Hi Chetan, can you please write this to our user list,
user@ignite.apache.org?
So that entire community can participate.
Thanks,
Pavel
On Fri, May 26, 2017 at 1:35 PM, Chetan D <cc...@gmail.com> wrote:
Hi Pavel Tupitsyn,
I have posted a comment in your blog as well (entity framework as ignite
.net store) regarding write behind using ignite.
I have been working on a project where i need to implement distributed
caching and i have been asked to look into grid gain.
This is is the first time i am working on caching and this is entirely new
topic for me.
The example which you have shared i was able to understand a little and the
sad part is even entity framework also i have never worked on.
It would be helpful if you can share me a simple example using ado.net
implementing read through, write through and write behind even a simple
table helps me understand the concept.
RE: Write behind using Grid Gain
Posted by Raymond Wilson <ra...@trimble.com>.
Denis,
Ah! Looks very interesting. Thanks for the pointer J
Raymond.
*From:* Denis Magda [mailto:dmagda@apache.org]
*Sent:* Wednesday, June 14, 2017 9:41 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
Raymond,
Then Ignite Persistent Store is exactly for your use case. Please refer to
this discussion on the dev list:
http://apache-ignite-developers.2346864.n4.nabble.com/GridGain-Donates-Persistent-Distributed-Store-To-ASF-Apache-Ignite-
td16788.html#a16838
Also it was covered a bit in that webinar:
https://www.youtube.com/watch?v=bDrGueQ16UQ
The store should be released by the community in the nearest.
—
Denis
On Jun 13, 2017, at 2:02 PM, Raymond Wilson <ra...@trimble.com>
wrote:
Hi Pavel,
It’s a little complicated. The system is essentially a DB in its own right;
actually it’s an IMDG a bit like Ignite, but developed 8 years ago to
fulfill a need we had.
Today, I am looking to modernize that system and rather than continuing to
build and maintain all the core ‘infrastructure’ features of an IMDG such
as clustering, messaging, enterprise caching etc, I am looking to see how
well Ignite fits by running a Proof of Concept project. It turns out it
fits quite well, largely because the architectural structure of both
systems (ie: IMDG) is well aligned in terms of the problems being solved.
The primary gap between the legacy system and IMDG is that IMDG does not
support persistence. The legacy system has a distributed cache that stores
objects that are aggregate collections (10’s of thousand’s) of relatively
simple spatial data records that are operated on by the clustered compute
engine. Sometimes billions of records need to be processed to satisfy a
single query. Your standard run of the mill SQL DB finds these sorts of
queries hard.
I suppose you could use another DB (MS-SQL, AWS:RDS etc) to store those
aggregate blobs, but it seems like a bit of a ‘miss-use case’ when what I’m
really after is a persistence/storage layer J
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Wednesday, June 14, 2017 1:59 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
Hi Raymond,
I think your use case fits well into traditional Ignite model of
write-through cache store with backing database.
Why do you want to avoid a DB? Do you plan to store data on disk directly
as a set of files?
Pavel
On Mon, Jun 12, 2017 at 2:14 AM, Raymond Wilson <ra...@trimble.com>
wrote:
Hi Pavel,
Thanks for the blog – it explains it quite well.
I have a slightly different use case where groups of records within a much
larger data set are clustered together for efficiency (ie: each of the
cached items in the Ignite grid cache has significant internal structure).
You can think of them as a large number of smallish files (a few Kb to a
few Mb), but file systems don’t like lots of small files.
I have a legacy implementation that houses these small files within a
single larger file, but wanted to know if there was a clean way of
supporting the same structure using the Ignite read/write through support,
perhaps with another system providing relatively transparent persistency
semantics but which does not use a DB to store the data.
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Saturday, May 27, 2017 5:03 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
I've decided to write a blog post, since this topic seems to be in demand:
https://ptupitsyn.github.io/Ado-Net-Cache-Store/
Code:
https://github.com/ptupitsyn/ignite-net-examples/tree/master/AdoNetCacheStore
Let me know if this helps!
On Fri, May 26, 2017 at 3:50 PM, Chetan D <cc...@gmail.com> wrote:
Thank you Pavel.
waiting for your response.
On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <pt...@apache.org>
wrote:
To give everyone context, this is not about GridGain, but about Apache
Ignite.
The blog post in question is
https://ptupitsyn.github.io/Entity-Framework-Cache-Store/
Chetan, I'll prepare an example with Ignite 2.0 / ado.net and post it some
time later.
Pavel
On Fri, May 26, 2017 at 2:32 PM, Chetan D <cc...@gmail.com> wrote:
++ User List
any help much appreciated.
Thanks And Regards
Chetan D
---------- Forwarded message ----------
From: *Pavel Tupitsyn* <pt...@apache.org>
Date: Fri, May 26, 2017 at 4:38 PM
Subject: Re: Write behind using Grid Gain
To: Chetan D <cc...@gmail.com>
Hi Chetan, can you please write this to our user list,
user@ignite.apache.org?
So that entire community can participate.
Thanks,
Pavel
On Fri, May 26, 2017 at 1:35 PM, Chetan D <cc...@gmail.com> wrote:
Hi Pavel Tupitsyn,
I have posted a comment in your blog as well (entity framework as ignite
.net store) regarding write behind using ignite.
I have been working on a project where i need to implement distributed
caching and i have been asked to look into grid gain.
This is is the first time i am working on caching and this is entirely new
topic for me.
The example which you have shared i was able to understand a little and the
sad part is even entity framework also i have never worked on.
It would be helpful if you can share me a simple example using ado.net
implementing
read through, write through and write behind even a simple table helps me
understand the concept.
Re: Write behind using Grid Gain
Posted by Denis Magda <dm...@apache.org>.
Raymond,
Then Ignite Persistent Store is exactly for your use case. Please refer to this discussion on the dev list:
http://apache-ignite-developers.2346864.n4.nabble.com/GridGain-Donates-Persistent-Distributed-Store-To-ASF-Apache-Ignite-td16788.html#a16838
Also it was covered a bit in that webinar:
https://www.youtube.com/watch?v=bDrGueQ16UQ
The store should be released by the community in the nearest.
—
Denis
> On Jun 13, 2017, at 2:02 PM, Raymond Wilson <ra...@trimble.com> wrote:
>
> Hi Pavel,
>
> It’s a little complicated. The system is essentially a DB in its own right; actually it’s an IMDG a bit like Ignite, but developed 8 years ago to fulfill a need we had.
>
> Today, I am looking to modernize that system and rather than continuing to build and maintain all the core ‘infrastructure’ features of an IMDG such as clustering, messaging, enterprise caching etc, I am looking to see how well Ignite fits by running a Proof of Concept project. It turns out it fits quite well, largely because the architectural structure of both systems (ie: IMDG) is well aligned in terms of the problems being solved.
>
> The primary gap between the legacy system and IMDG is that IMDG does not support persistence. The legacy system has a distributed cache that stores objects that are aggregate collections (10’s of thousand’s) of relatively simple spatial data records that are operated on by the clustered compute engine. Sometimes billions of records need to be processed to satisfy a single query. Your standard run of the mill SQL DB finds these sorts of queries hard.
>
> I suppose you could use another DB (MS-SQL, AWS:RDS etc) to store those aggregate blobs, but it seems like a bit of a ‘miss-use case’ when what I’m really after is a persistence/storage layer J
>
> Thanks,
> Raymond.
>
>
> From: Pavel Tupitsyn [mailto:ptupitsyn@apache.org <ma...@apache.org>]
> Sent: Wednesday, June 14, 2017 1:59 AM
> To: user@ignite.apache.org <ma...@ignite.apache.org>
> Subject: Re: Write behind using Grid Gain
>
> Hi Raymond,
>
> I think your use case fits well into traditional Ignite model of write-through cache store with backing database.
> Why do you want to avoid a DB? Do you plan to store data on disk directly as a set of files?
>
> Pavel
>
> On Mon, Jun 12, 2017 at 2:14 AM, Raymond Wilson <raymond_wilson@trimble.com <ma...@trimble.com>> wrote:
> Hi Pavel,
>
> Thanks for the blog – it explains it quite well.
>
> I have a slightly different use case where groups of records within a much larger data set are clustered together for efficiency (ie: each of the cached items in the Ignite grid cache has significant internal structure). You can think of them as a large number of smallish files (a few Kb to a few Mb), but file systems don’t like lots of small files.
>
> I have a legacy implementation that houses these small files within a single larger file, but wanted to know if there was a clean way of supporting the same structure using the Ignite read/write through support, perhaps with another system providing relatively transparent persistency semantics but which does not use a DB to store the data.
>
> Thanks,
> Raymond.
>
> From: Pavel Tupitsyn [mailto:ptupitsyn@apache.org <ma...@apache.org>]
> Sent: Saturday, May 27, 2017 5:03 AM
> To: user@ignite.apache.org <ma...@ignite.apache.org>
>
> Subject: Re: Write behind using Grid Gain
>
> I've decided to write a blog post, since this topic seems to be in demand:
> https://ptupitsyn.github.io/Ado-Net-Cache-Store/ <https://ptupitsyn.github.io/Ado-Net-Cache-Store/>
>
> Code:
> https://github.com/ptupitsyn/ignite-net-examples/tree/master/AdoNetCacheStore <https://github.com/ptupitsyn/ignite-net-examples/tree/master/AdoNetCacheStore>
>
> Let me know if this helps!
>
> On Fri, May 26, 2017 at 3:50 PM, Chetan D <cchetands@gmail.com <ma...@gmail.com>> wrote:
> Thank you Pavel.
>
> waiting for your response.
>
> On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <ptupitsyn@apache.org <ma...@apache.org>> wrote:
> To give everyone context, this is not about GridGain, but about Apache Ignite.
> The blog post in question is https://ptupitsyn.github.io/Entity-Framework-Cache-Store/ <https://ptupitsyn.github.io/Entity-Framework-Cache-Store/>
>
> Chetan, I'll prepare an example with Ignite 2.0 / ado.net <http://ado.net/> and post it some time later.
>
> Pavel
>
> On Fri, May 26, 2017 at 2:32 PM, Chetan D <cchetands@gmail.com <ma...@gmail.com>> wrote:
> ++ User List
>
> any help much appreciated.
>
> Thanks And Regards
> Chetan D
> ---------- Forwarded message ----------
> From: Pavel Tupitsyn <ptupitsyn@apache.org <ma...@apache.org>>
> Date: Fri, May 26, 2017 at 4:38 PM
> Subject: Re: Write behind using Grid Gain
> To: Chetan D <cchetands@gmail.com <ma...@gmail.com>>
>
> Hi Chetan, can you please write this to our user list, user@ignite.apache.org <ma...@ignite.apache.org>?
> So that entire community can participate.
>
> Thanks,
> Pavel
>
> On Fri, May 26, 2017 at 1:35 PM, Chetan D <cchetands@gmail.com <ma...@gmail.com>> wrote:
> Hi Pavel Tupitsyn,
>
> I have posted a comment in your blog as well (entity framework as ignite .net store) regarding write behind using ignite.
>
> I have been working on a project where i need to implement distributed caching and i have been asked to look into grid gain.
>
> This is is the first time i am working on caching and this is entirely new topic for me.
>
> The example which you have shared i was able to understand a little and the sad part is even entity framework also i have never worked on.
>
> It would be helpful if you can share me a simple example using ado.net <http://ado.net/> implementing read through, write through and write behind even a simple table helps me understand the concept.
RE: Write behind using Grid Gain
Posted by Raymond Wilson <ra...@trimble.com>.
Hi Pavel,
It’s a little complicated. The system is essentially a DB in its own right;
actually it’s an IMDG a bit like Ignite, but developed 8 years ago to
fulfill a need we had.
Today, I am looking to modernize that system and rather than continuing to
build and maintain all the core ‘infrastructure’ features of an IMDG such
as clustering, messaging, enterprise caching etc, I am looking to see how
well Ignite fits by running a Proof of Concept project. It turns out it
fits quite well, largely because the architectural structure of both
systems (ie: IMDG) is well aligned in terms of the problems being solved.
The primary gap between the legacy system and IMDG is that IMDG does not
support persistence. The legacy system has a distributed cache that stores
objects that are aggregate collections (10’s of thousand’s) of relatively
simple spatial data records that are operated on by the clustered compute
engine. Sometimes billions of records need to be processed to satisfy a
single query. Your standard run of the mill SQL DB finds these sorts of
queries hard.
I suppose you could use another DB (MS-SQL, AWS:RDS etc) to store those
aggregate blobs, but it seems like a bit of a ‘miss-use case’ when what I’m
really after is a persistence/storage layer J
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Wednesday, June 14, 2017 1:59 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
Hi Raymond,
I think your use case fits well into traditional Ignite model of
write-through cache store with backing database.
Why do you want to avoid a DB? Do you plan to store data on disk directly
as a set of files?
Pavel
On Mon, Jun 12, 2017 at 2:14 AM, Raymond Wilson <ra...@trimble.com>
wrote:
Hi Pavel,
Thanks for the blog – it explains it quite well.
I have a slightly different use case where groups of records within a much
larger data set are clustered together for efficiency (ie: each of the
cached items in the Ignite grid cache has significant internal structure).
You can think of them as a large number of smallish files (a few Kb to a
few Mb), but file systems don’t like lots of small files.
I have a legacy implementation that houses these small files within a
single larger file, but wanted to know if there was a clean way of
supporting the same structure using the Ignite read/write through support,
perhaps with another system providing relatively transparent persistency
semantics but which does not use a DB to store the data.
Thanks,
Raymond.
*From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
*Sent:* Saturday, May 27, 2017 5:03 AM
*To:* user@ignite.apache.org
*Subject:* Re: Write behind using Grid Gain
I've decided to write a blog post, since this topic seems to be in demand:
https://ptupitsyn.github.io/Ado-Net-Cache-Store/
Code:
https://github.com/ptupitsyn/ignite-net-examples/tree/master/AdoNetCacheStore
Let me know if this helps!
On Fri, May 26, 2017 at 3:50 PM, Chetan D <cc...@gmail.com> wrote:
Thank you Pavel.
waiting for your response.
On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <pt...@apache.org>
wrote:
To give everyone context, this is not about GridGain, but about Apache
Ignite.
The blog post in question is
https://ptupitsyn.github.io/Entity-Framework-Cache-Store/
Chetan, I'll prepare an example with Ignite 2.0 / ado.net and post it some
time later.
Pavel
On Fri, May 26, 2017 at 2:32 PM, Chetan D <cc...@gmail.com> wrote:
++ User List
any help much appreciated.
Thanks And Regards
Chetan D
---------- Forwarded message ----------
From: *Pavel Tupitsyn* <pt...@apache.org>
Date: Fri, May 26, 2017 at 4:38 PM
Subject: Re: Write behind using Grid Gain
To: Chetan D <cc...@gmail.com>
Hi Chetan, can you please write this to our user list,
user@ignite.apache.org?
So that entire community can participate.
Thanks,
Pavel
On Fri, May 26, 2017 at 1:35 PM, Chetan D <cc...@gmail.com> wrote:
Hi Pavel Tupitsyn,
I have posted a comment in your blog as well (entity framework as ignite
.net store) regarding write behind using ignite.
I have been working on a project where i need to implement distributed
caching and i have been asked to look into grid gain.
This is is the first time i am working on caching and this is entirely new
topic for me.
The example which you have shared i was able to understand a little and the
sad part is even entity framework also i have never worked on.
It would be helpful if you can share me a simple example using ado.net
implementing read through, write through and write behind even a simple
table helps me understand the concept.
Re: Write behind using Grid Gain
Posted by Pavel Tupitsyn <pt...@apache.org>.
Hi Raymond,
I think your use case fits well into traditional Ignite model of
write-through cache store with backing database.
Why do you want to avoid a DB? Do you plan to store data on disk directly
as a set of files?
Pavel
On Mon, Jun 12, 2017 at 2:14 AM, Raymond Wilson <ra...@trimble.com>
wrote:
> Hi Pavel,
>
>
>
> Thanks for the blog – it explains it quite well.
>
>
>
> I have a slightly different use case where groups of records within a much
> larger data set are clustered together for efficiency (ie: each of the
> cached items in the Ignite grid cache has significant internal structure).
> You can think of them as a large number of smallish files (a few Kb to a
> few Mb), but file systems don’t like lots of small files.
>
>
>
> I have a legacy implementation that houses these small files within a
> single larger file, but wanted to know if there was a clean way of
> supporting the same structure using the Ignite read/write through support,
> perhaps with another system providing relatively transparent persistency
> semantics but which does not use a DB to store the data.
>
>
>
> Thanks,
>
> Raymond.
>
>
>
> *From:* Pavel Tupitsyn [mailto:ptupitsyn@apache.org]
> *Sent:* Saturday, May 27, 2017 5:03 AM
> *To:* user@ignite.apache.org
>
> *Subject:* Re: Write behind using Grid Gain
>
>
>
> I've decided to write a blog post, since this topic seems to be in demand:
>
> https://ptupitsyn.github.io/Ado-Net-Cache-Store/
>
>
>
> Code:
>
> https://github.com/ptupitsyn/ignite-net-examples/tree/
> master/AdoNetCacheStore
>
>
>
> Let me know if this helps!
>
>
>
> On Fri, May 26, 2017 at 3:50 PM, Chetan D <cc...@gmail.com> wrote:
>
> Thank you Pavel.
>
>
>
> waiting for your response.
>
>
>
> On Fri, May 26, 2017 at 6:03 PM, Pavel Tupitsyn <pt...@apache.org>
> wrote:
>
> To give everyone context, this is not about GridGain, but about Apache
> Ignite.
>
> The blog post in question is https://ptupitsyn.github.
> io/Entity-Framework-Cache-Store/
>
>
>
> Chetan, I'll prepare an example with Ignite 2.0 / ado.net and post it
> some time later.
>
>
>
> Pavel
>
>
>
> On Fri, May 26, 2017 at 2:32 PM, Chetan D <cc...@gmail.com> wrote:
>
> ++ User List
>
>
>
> any help much appreciated.
>
>
>
> Thanks And Regards
>
> Chetan D
>
> ---------- Forwarded message ----------
> From: *Pavel Tupitsyn* <pt...@apache.org>
> Date: Fri, May 26, 2017 at 4:38 PM
> Subject: Re: Write behind using Grid Gain
> To: Chetan D <cc...@gmail.com>
>
> Hi Chetan, can you please write this to our user list,
> user@ignite.apache.org?
>
> So that entire community can participate.
>
>
>
> Thanks,
>
> Pavel
>
>
>
> On Fri, May 26, 2017 at 1:35 PM, Chetan D <cc...@gmail.com> wrote:
>
> Hi Pavel Tupitsyn,
>
>
>
> I have posted a comment in your blog as well (entity framework as ignite
> .net store) regarding write behind using ignite.
>
>
>
> I have been working on a project where i need to implement distributed
> caching and i have been asked to look into grid gain.
>
>
>
> This is is the first time i am working on caching and this is entirely new
> topic for me.
>
>
>
> The example which you have shared i was able to understand a little and
> the sad part is even entity framework also i have never worked on.
>
>
>
> It would be helpful if you can share me a simple example using ado.net
> implementing read through, write through and write behind even a simple
> table helps me understand the concept.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>