You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Clay Teahouse <cl...@gmail.com> on 2018/12/31 13:01:05 UTC
ignite questions
Hello All,
I am new to ignite and have several general questions. I'd appreciate your
feedback.
1) Cache groups: according to the ignite documentation, cache groups help
with scaling and performance but might hurt reads. Where is the balance?
2) Capacity planning: If I reading the docs correctly, with native
persistence enabled, you do not need to specify cache eviction. If so,
assuming I have data and compute affinity enabled, how do I size my nodes,
to make sure my data stays in cache, considering volume discrepancy in
different class of data? Say for example, I have the data for Canada and
India, with India data being 10 times the data for Canada. How do I size my
nodes to make sure the last month data for India and Canada stay in cache?
3) Data pin to cache: How do I make sure certain data never gets evicted
(with native persistence enabled)? For example, I want my dimension data to
always stay in cache.
4) Read through: If I am using native persistence, do I need to explicitly
load the cache, once the data is on disk and no longer in cache, or doing a
read to the data on the disk, will load the cache? If yes, is that true
about SQL select as well? Is this possible with 3rd party persistence as
well, say, postgresql.
5) Service chaining: Is there an example of service chaining that you can
point me to?
6) How do I implement service pipelining in apache ignite? Would continuous
query be the mechanism? Any examples?
7) Streaming: Are there examples on how to define watermarks, i.e., input
completeness with regard to the event timestamp?
thank you,
Clay
Re: ignite questions
Posted by Clay Teahouse <cl...@gmail.com>.
I'd have at least 80-90 caches, assuming each cache corresponds to a table.
Any feedback regarding other questions?
thanks,
Clay
On Mon, Dec 31, 2018 at 9:58 AM Mikael <mi...@telia.com> wrote:
> Hi!
>
> You have to try, if you just have a few caches (<10) you may not need to
> go for any cache groups at all but the more caches you have the more the
> need for cache group pops up, it will create lots of file handles and
> use lots of memory that can be kept under control with cache groups.
>
> Mikael
>
> Den 2018-12-31 kl. 14:01, skrev Clay Teahouse:
> > Hello All,
> > I am new to ignite and have several general questions. I'd appreciate
> > your feedback.
> >
> > 1) Cache groups: according to the ignite documentation, cache groups
> > help with scaling and performance but might hurt reads. Where is the
> > balance?
> >
> > 2) Capacity planning: If I reading the docs correctly, with native
> > persistence enabled, you do not need to specify cache eviction. If so,
> > assuming I have data and compute affinity enabled, how do I size my
> > nodes, to make sure my data stays in cache, considering volume
> > discrepancy in different class of data? Say for example, I have the
> > data for Canada and India, with India data being 10 times the data for
> > Canada. How do I size my nodes to make sure the last month data for
> > India and Canada stay in cache?
> >
> > 3) Data pin to cache: How do I make sure certain data never gets
> > evicted (with native persistence enabled)? For example, I want my
> > dimension data to always stay in cache.
> >
> > 4) Read through: If I am using native persistence, do I need to
> > explicitly load the cache, once the data is on disk and no longer in
> > cache, or doing a read to the data on the disk, will load the cache?
> > If yes, is that true about SQL select as well? Is this possible with
> > 3rd party persistence as well, say, postgresql.
> >
> > 5) Service chaining: Is there an example of service chaining that you
> > can point me to?
> >
> > 6) How do I implement service pipelining in apache ignite? Would
> > continuous query be the mechanism? Any examples?
> >
> > 7) Streaming: Are there examples on how to define watermarks, i.e.,
> > input completeness with regard to the event timestamp?
> >
> >
> >
> > thank you,
> > Clay
>
Re: ignite questions
Posted by Mikael <mi...@telia.com>.
Hi!
You have to try, if you just have a few caches (<10) you may not need to
go for any cache groups at all but the more caches you have the more the
need for cache group pops up, it will create lots of file handles and
use lots of memory that can be kept under control with cache groups.
Mikael
Den 2018-12-31 kl. 14:01, skrev Clay Teahouse:
> Hello All,
> I am new to ignite and have several general questions. I'd appreciate
> your feedback.
>
> 1) Cache groups: according to the ignite documentation, cache groups
> help with scaling and performance but might hurt reads. Where is the
> balance?
>
> 2) Capacity planning: If I reading the docs correctly, with native
> persistence enabled, you do not need to specify cache eviction. If so,
> assuming I have data and compute affinity enabled, how do I size my
> nodes, to make sure my data stays in cache, considering volume
> discrepancy in different class of data? Say for example, I have the
> data for Canada and India, with India data being 10 times the data for
> Canada. How do I size my nodes to make sure the last month data for
> India and Canada stay in cache?
>
> 3) Data pin to cache: How do I make sure certain data never gets
> evicted (with native persistence enabled)? For example, I want my
> dimension data to always stay in cache.
>
> 4) Read through: If I am using native persistence, do I need to
> explicitly load the cache, once the data is on disk and no longer in
> cache, or doing a read to the data on the disk, will load the cache?
> If yes, is that true about SQL select as well? Is this possible with
> 3rd party persistence as well, say, postgresql.
>
> 5) Service chaining: Is there an example of service chaining that you
> can point me to?
>
> 6) How do I implement service pipelining in apache ignite? Would
> continuous query be the mechanism? Any examples?
>
> 7) Streaming: Are there examples on how to define watermarks, i.e.,
> input completeness with regard to the event timestamp?
>
>
>
> thank you,
> Clay
Re: ignite questions
Posted by Clay Teahouse <cl...@gmail.com>.
Thanks for everyone's feedback regarding the capacity planning question. My
main objective here is to size my servers accordingly and keep the related
data on the same server, as much as possible. It seems a custom affinity
function that takes into account the server classes (defined based on
capacity) is the potential solution as suggested. Naveen -- I will keep in
mind your suggestion as well, regarding partitioning the huge data sets.
thanks.
Any feedback regarding my other questions:
Data pin to cache: How do I make sure certain data never gets evicted (with
native persistence enabled)? For example, I want my dimension data to
always stay in cache.
How do I implement service pipelining in apache ignite? Would continuous
query be the mechanism? Any examples?
Streaming: Are there examples on how to define watermarks, i.e., input
completeness with regard to the event timestamp?
On Fri, Jan 4, 2019 at 3:33 AM Naveen <na...@gmail.com> wrote:
> Regarding your question on capacity planning
>
> Not sure we have any work around to have data equally getting distributed
> to
> fulfill your requirement technically and do the sizing. But
> non-technically,
> you can change your design to include state as well as part of your
> affinity
> key along with the country and rest of the fields you have, so that data
> pertaining to India can be segregated into chunks of data which is state
> wise and if you see statewide data is definitely not so huge compared to
> country as a whole and same time relevant data is also getting stored on a
> single node. Your design also should permit this to keep state name as
> part
> of your affinity key to resolve this use case.
>
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: ignite questions
Posted by Naveen <na...@gmail.com>.
Regarding your question on capacity planning
Not sure we have any work around to have data equally getting distributed to
fulfill your requirement technically and do the sizing. But non-technically,
you can change your design to include state as well as part of your affinity
key along with the country and rest of the fields you have, so that data
pertaining to India can be segregated into chunks of data which is state
wise and if you see statewide data is definitely not so huge compared to
country as a whole and same time relevant data is also getting stored on a
single node. Your design also should permit this to keep state name as part
of your affinity key to resolve this use case.
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: ignite questions
Posted by Denis Magda <dm...@apache.org>.
Yes, a custom affinity function is what you need to control entries
distribution across physical machines. It's feasible to do. Worked with one
of Ignite customers who did something similar for their needs - the code is
not open sourced.
--
Denis
On Wed, Jan 2, 2019 at 10:17 AM Mikael <mi...@telia.com> wrote:
> Hi!
>
> By default you cannot assign a specific affinity key to a specific node
> but I think that could be done with a custom affinity function, you can do
> pretty much whatever you want with that, for example set an attribute in
> the XML file and use that to match with a specific affinity key value, so a
> node with attribute x will be assigned all affinity keys with value y.
>
> I never tried it but I do not see any reason why it would not work.
>
> Mikael
>
>
> Den 2019-01-02 kl. 17:13, skrev Clay Teahouse:
>
> Thanks Mikael.
>
> I did come across that link before, but I am not sure it addresses my
> concern. I want to see how I need I size my physical VMs based on affinity
> keys. How would I say for India affinity key use this super size VM and for
> others use the other smaller ones, so the data doesn't get shuffled around?
> Maybe, there is no way, and I just have to wait for ignite to rebalance the
> partitions and fit things where they should be based on the affinity key.
>
> On Wed, Jan 2, 2019 at 8:32 AM Mikael <mi...@telia.com> wrote:
>
>> You can find some information about capacity planning here:
>>
>> https://apacheignite.readme.io/docs/capacity-planning
>>
>> About your India example you can use affinity keys to keep data together
>> in groups to avoid network traffic.
>>
>> https://apacheignite.readme.io/docs/affinity-collocation
>>
>> Mikael
>> Den 2019-01-02 kl. 14:44, skrev Clay Teahouse:
>>
>> Thanks Naveen.
>>
>> -- Cache Groups: When would I start considering cache groups, if my
>> system is growing, and sooner or later I will have to add to my caches and
>> I need to know 1) should I starting grouping now (I'd think yes), 2) if no,
>> when, what number of caches?
>> -- Capacity Planning: So, there is no guidelines on how to size the nodes
>> and the physical storage nodes reside on? How do I make sure all the
>> related data fit the same VM? It can't be the case that I have to come up
>> with 100s of super size VMs just because I have one instance with a huge
>> set of entries. For example, if I have millions of entries for India and
>> only a few for other countries, how do I make sure all the India related
>> data fits the same VM (to avoid the network) and have the data for all the
>> small countries fit on the same VM?
>> -- Pinning the data to cache: the data pinned to on-heap cache does not
>> get evicted from the memory? I want to see if there is something similar to
>> Oracle's memory pinning.
>> -- Read through: How do I know if something on cache or disk (using
>> native persistence)?
>> 5) Service chaining: Is there an example of service chaining that you can
>> point me to?
>>
>> 6) How do I implement service pipelining in apache ignite? Would
>> continuous query be the mechanism? Any examples?
>>
>> 7) Streaming: Are there examples on how to define watermarks, i.e., input
>> completeness with regard to the event timestamp?
>>
>> thank you
>> Clay
>>
>> On Tue, Jan 1, 2019 at 11:29 PM Naveen <na...@gmail.com> wrote:
>>
>>> Hello
>>> Couple of things I would like to with my experience
>>>
>>> 1. Cache Groups : Around 100 caches, I do not think we need to go for
>>> Cache
>>> groups, as you mentioned cache groups will have impact on you
>>> read/writes.
>>> However, changing the partition count to 128 from default 1024 would
>>> improve
>>> your cluster restart.
>>>
>>> 2. I doubt if Ignite has any settings we have for this.
>>>
>>> 3. The only I can think of is to keep the data in on-heap if the data
>>> size
>>> is not so huge.
>>>
>>> 4. Read through, with native persistence enabled, doing a read to the
>>> disk
>>> will load the cache. But the read is much slower compared with read from
>>> RAM, by default it does not pre-load the data. If you want to avoid this
>>> you
>>> can pre-load the data programatically and load Memory, good for even SQL
>>> SELECT as well. But with the 3rd party persistence, we need to pre-load
>>> the
>>> data to make your read work for SQL SELECT.
>>>
>>> Thanks
>>> Naveen
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
Re: ignite questions
Posted by Mikael <mi...@telia.com>.
Hi!
By default you cannot assign a specific affinity key to a specific node
but I think that could be done with a custom affinity function, you can
do pretty much whatever you want with that, for example set an attribute
in the XML file and use that to match with a specific affinity key
value, so a node with attribute x will be assigned all affinity keys
with value y.
I never tried it but I do not see any reason why it would not work.
Mikael
Den 2019-01-02 kl. 17:13, skrev Clay Teahouse:
> Thanks Mikael.
>
> I did come across that link before, but I am not sure it addresses my
> concern. I want to see how I need I size my physical VMs based on
> affinity keys. How would I say for India affinity key use this super
> size VM and for others use the other smaller ones, so the data doesn't
> get shuffled around? Maybe, there is no way, and I just have to wait
> for ignite to rebalance the partitions and fit things where they
> should be based on the affinity key.
>
> On Wed, Jan 2, 2019 at 8:32 AM Mikael <mikael-aronsson@telia.com
> <ma...@telia.com>> wrote:
>
> You can find some information about capacity planning here:
>
> https://apacheignite.readme.io/docs/capacity-planning
>
> About your India example you can use affinity keys to keep data
> together in groups to avoid network traffic.
>
> https://apacheignite.readme.io/docs/affinity-collocation
>
> Mikael
>
> Den 2019-01-02 kl. 14:44, skrev Clay Teahouse:
>> Thanks Naveen.
>>
>> -- Cache Groups: When would I start considering cache groups, if
>> my system is growing, and sooner or later I will have to add to
>> my caches and I need to know 1) should I starting grouping now
>> (I'd think yes), 2) if no, when, what number of caches?
>> -- Capacity Planning: So, there is no guidelines on how to size
>> the nodes and the physical storage nodes reside on? How do I make
>> sure all the related data fit the same VM? It can't be the case
>> that I have to come up with 100s of super size VMs just because I
>> have one instance with a huge set of entries. For example, if I
>> have millions of entries for India and only a few for other
>> countries, how do I make sure all the India related data fits the
>> same VM (to avoid the network) and have the data for all the
>> small countries fit on the same VM?
>> -- Pinning the data to cache: the data pinned to on-heap cache
>> does not get evicted from the memory? I want to see if there is
>> something similar to Oracle's memory pinning.
>> -- Read through: How do I know if something on cache or disk
>> (using native persistence)?
>> 5) Service chaining: Is there an example of service chaining that
>> you can point me to?
>>
>> 6) How do I implement service pipelining in apache ignite? Would
>> continuous query be the mechanism? Any examples?
>>
>> 7) Streaming: Are there examples on how to define watermarks,
>> i.e., input completeness with regard to the event timestamp?
>>
>> thank you
>> Clay
>>
>> On Tue, Jan 1, 2019 at 11:29 PM Naveen <naveen.bandaru@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> Hello
>> Couple of things I would like to with my experience
>>
>> 1. Cache Groups : Around 100 caches, I do not think we need
>> to go for Cache
>> groups, as you mentioned cache groups will have impact on you
>> read/writes.
>> However, changing the partition count to 128 from default
>> 1024 would improve
>> your cluster restart.
>>
>> 2. I doubt if Ignite has any settings we have for this.
>>
>> 3. The only I can think of is to keep the data in on-heap if
>> the data size
>> is not so huge.
>>
>> 4. Read through, with native persistence enabled, doing a
>> read to the disk
>> will load the cache. But the read is much slower compared
>> with read from
>> RAM, by default it does not pre-load the data. If you want to
>> avoid this you
>> can pre-load the data programatically and load Memory, good
>> for even SQL
>> SELECT as well. But with the 3rd party persistence, we need
>> to pre-load the
>> data to make your read work for SQL SELECT.
>>
>> Thanks
>> Naveen
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
Re: ignite questions
Posted by Clay Teahouse <cl...@gmail.com>.
Thanks Mikael.
I did come across that link before, but I am not sure it addresses my
concern. I want to see how I need I size my physical VMs based on affinity
keys. How would I say for India affinity key use this super size VM and for
others use the other smaller ones, so the data doesn't get shuffled around?
Maybe, there is no way, and I just have to wait for ignite to rebalance the
partitions and fit things where they should be based on the affinity key.
On Wed, Jan 2, 2019 at 8:32 AM Mikael <mi...@telia.com> wrote:
> You can find some information about capacity planning here:
>
> https://apacheignite.readme.io/docs/capacity-planning
>
> About your India example you can use affinity keys to keep data together
> in groups to avoid network traffic.
>
> https://apacheignite.readme.io/docs/affinity-collocation
>
> Mikael
> Den 2019-01-02 kl. 14:44, skrev Clay Teahouse:
>
> Thanks Naveen.
>
> -- Cache Groups: When would I start considering cache groups, if my system
> is growing, and sooner or later I will have to add to my caches and I need
> to know 1) should I starting grouping now (I'd think yes), 2) if no, when,
> what number of caches?
> -- Capacity Planning: So, there is no guidelines on how to size the nodes
> and the physical storage nodes reside on? How do I make sure all the
> related data fit the same VM? It can't be the case that I have to come up
> with 100s of super size VMs just because I have one instance with a huge
> set of entries. For example, if I have millions of entries for India and
> only a few for other countries, how do I make sure all the India related
> data fits the same VM (to avoid the network) and have the data for all the
> small countries fit on the same VM?
> -- Pinning the data to cache: the data pinned to on-heap cache does not
> get evicted from the memory? I want to see if there is something similar to
> Oracle's memory pinning.
> -- Read through: How do I know if something on cache or disk (using native
> persistence)?
> 5) Service chaining: Is there an example of service chaining that you can
> point me to?
>
> 6) How do I implement service pipelining in apache ignite? Would
> continuous query be the mechanism? Any examples?
>
> 7) Streaming: Are there examples on how to define watermarks, i.e., input
> completeness with regard to the event timestamp?
>
> thank you
> Clay
>
> On Tue, Jan 1, 2019 at 11:29 PM Naveen <na...@gmail.com> wrote:
>
>> Hello
>> Couple of things I would like to with my experience
>>
>> 1. Cache Groups : Around 100 caches, I do not think we need to go for
>> Cache
>> groups, as you mentioned cache groups will have impact on you read/writes.
>> However, changing the partition count to 128 from default 1024 would
>> improve
>> your cluster restart.
>>
>> 2. I doubt if Ignite has any settings we have for this.
>>
>> 3. The only I can think of is to keep the data in on-heap if the data size
>> is not so huge.
>>
>> 4. Read through, with native persistence enabled, doing a read to the disk
>> will load the cache. But the read is much slower compared with read from
>> RAM, by default it does not pre-load the data. If you want to avoid this
>> you
>> can pre-load the data programatically and load Memory, good for even SQL
>> SELECT as well. But with the 3rd party persistence, we need to pre-load
>> the
>> data to make your read work for SQL SELECT.
>>
>> Thanks
>> Naveen
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: ignite questions
Posted by Mikael <mi...@telia.com>.
You can find some information about capacity planning here:
https://apacheignite.readme.io/docs/capacity-planning
About your India example you can use affinity keys to keep data together
in groups to avoid network traffic.
https://apacheignite.readme.io/docs/affinity-collocation
Mikael
Den 2019-01-02 kl. 14:44, skrev Clay Teahouse:
> Thanks Naveen.
>
> -- Cache Groups: When would I start considering cache groups, if my
> system is growing, and sooner or later I will have to add to my caches
> and I need to know 1) should I starting grouping now (I'd think yes),
> 2) if no, when, what number of caches?
> -- Capacity Planning: So, there is no guidelines on how to size the
> nodes and the physical storage nodes reside on? How do I make sure all
> the related data fit the same VM? It can't be the case that I have to
> come up with 100s of super size VMs just because I have one instance
> with a huge set of entries. For example, if I have millions of entries
> for India and only a few for other countries, how do I make sure all
> the India related data fits the same VM (to avoid the network) and
> have the data for all the small countries fit on the same VM?
> -- Pinning the data to cache: the data pinned to on-heap cache does
> not get evicted from the memory? I want to see if there is something
> similar to Oracle's memory pinning.
> -- Read through: How do I know if something on cache or disk (using
> native persistence)?
> 5) Service chaining: Is there an example of service chaining that you
> can point me to?
>
> 6) How do I implement service pipelining in apache ignite? Would
> continuous query be the mechanism? Any examples?
>
> 7) Streaming: Are there examples on how to define watermarks, i.e.,
> input completeness with regard to the event timestamp?
>
> thank you
> Clay
>
> On Tue, Jan 1, 2019 at 11:29 PM Naveen <naveen.bandaru@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hello
> Couple of things I would like to with my experience
>
> 1. Cache Groups : Around 100 caches, I do not think we need to go
> for Cache
> groups, as you mentioned cache groups will have impact on you
> read/writes.
> However, changing the partition count to 128 from default 1024
> would improve
> your cluster restart.
>
> 2. I doubt if Ignite has any settings we have for this.
>
> 3. The only I can think of is to keep the data in on-heap if the
> data size
> is not so huge.
>
> 4. Read through, with native persistence enabled, doing a read to
> the disk
> will load the cache. But the read is much slower compared with
> read from
> RAM, by default it does not pre-load the data. If you want to
> avoid this you
> can pre-load the data programatically and load Memory, good for
> even SQL
> SELECT as well. But with the 3rd party persistence, we need to
> pre-load the
> data to make your read work for SQL SELECT.
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: ignite questions
Posted by Clay Teahouse <cl...@gmail.com>.
Thanks Naveen.
-- Cache Groups: When would I start considering cache groups, if my system
is growing, and sooner or later I will have to add to my caches and I need
to know 1) should I starting grouping now (I'd think yes), 2) if no, when,
what number of caches?
-- Capacity Planning: So, there is no guidelines on how to size the nodes
and the physical storage nodes reside on? How do I make sure all the
related data fit the same VM? It can't be the case that I have to come up
with 100s of super size VMs just because I have one instance with a huge
set of entries. For example, if I have millions of entries for India and
only a few for other countries, how do I make sure all the India related
data fits the same VM (to avoid the network) and have the data for all the
small countries fit on the same VM?
-- Pinning the data to cache: the data pinned to on-heap cache does not get
evicted from the memory? I want to see if there is something similar to
Oracle's memory pinning.
-- Read through: How do I know if something on cache or disk (using native
persistence)?
5) Service chaining: Is there an example of service chaining that you can
point me to?
6) How do I implement service pipelining in apache ignite? Would continuous
query be the mechanism? Any examples?
7) Streaming: Are there examples on how to define watermarks, i.e., input
completeness with regard to the event timestamp?
thank you
Clay
On Tue, Jan 1, 2019 at 11:29 PM Naveen <na...@gmail.com> wrote:
> Hello
> Couple of things I would like to with my experience
>
> 1. Cache Groups : Around 100 caches, I do not think we need to go for Cache
> groups, as you mentioned cache groups will have impact on you read/writes.
> However, changing the partition count to 128 from default 1024 would
> improve
> your cluster restart.
>
> 2. I doubt if Ignite has any settings we have for this.
>
> 3. The only I can think of is to keep the data in on-heap if the data size
> is not so huge.
>
> 4. Read through, with native persistence enabled, doing a read to the disk
> will load the cache. But the read is much slower compared with read from
> RAM, by default it does not pre-load the data. If you want to avoid this
> you
> can pre-load the data programatically and load Memory, good for even SQL
> SELECT as well. But with the 3rd party persistence, we need to pre-load the
> data to make your read work for SQL SELECT.
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: ignite questions
Posted by Naveen <na...@gmail.com>.
Hello
Couple of things I would like to with my experience
1. Cache Groups : Around 100 caches, I do not think we need to go for Cache
groups, as you mentioned cache groups will have impact on you read/writes.
However, changing the partition count to 128 from default 1024 would improve
your cluster restart.
2. I doubt if Ignite has any settings we have for this.
3. The only I can think of is to keep the data in on-heap if the data size
is not so huge.
4. Read through, with native persistence enabled, doing a read to the disk
will load the cache. But the read is much slower compared with read from
RAM, by default it does not pre-load the data. If you want to avoid this you
can pre-load the data programatically and load Memory, good for even SQL
SELECT as well. But with the 3rd party persistence, we need to pre-load the
data to make your read work for SQL SELECT.
Thanks
Naveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/