Queries on setting up real-time analytics platform on ignite.

[Ed Smith <ph...@binaryflux.xyz>]

Hi,

I needs info on few things before i start using ignite, my system is a big
data , high eps (500k eps) , analytics system (gnn). I am right now
inclined towards using native persistence.

1. Is it possible to create hot ,warm, cold and frozen data sets ?
considering hot means most user data set and so on.

2. If i choose to use key value as storage , any api or language available
to run queries over this data , specially aggregations (count, sum etc.)

3 Will i be able to create partitions based on some logical data types.

4. How do i setup a multi tenant architecture, specially where one user can
have access to multiple tenants.

5. Is it possible to save data as tree in ignite ?


I will appreciate if you could help me determine answers to these questions
as i need to finalise on which framework to use in order to run above kind
of system.


Thanks
ED.

Re: Queries on setting up real-time analytics platform on ignite.

[Denis Magda <dm...@apache.org>]

Hi Ed,

1. Yes, that's possible and you do this in Ignite by configuring several
data regions
<https://apacheignite.readme.io/docs/memory-configuration#section-data-regions>.
All the caches/tables with hot data will be mapped to data regions that are
given most of the memory resources, while caches/tables with cold data can
reside in data regions with little memory space.

2. Even though Ignite supports key-value APIs and can be used as a
key-value store, it's storage engine is classified as a B-tree-based
storage engine
<https://www.gridgain.com/docs/latest/developers-guide/memory-centric-storage#memory-architecture>.
Having said that, you can use any APIs to process Ignite records.

3. Yes, that's doable. This page
<https://www.gridgain.com/docs/latest/developers-guide/configuring-caches/managing-data-distribution>
covers the most straightforward approaches for data distribution management.

4. Ignite doesn't support multi-tenancy out-of-the-box. So, you can build
it in house and plugin into Ignite or use enterprise solutions
<https://www.gridgain.com/docs/latest/administrators-guide/security/multi-tenancy>
.

5. Yes, you can do that. But it might be expensive to update records that
store tree objects. Probably, you don't even need that as long as Ignite is
a B-tree-based storage engine.

-
Denis


On Thu, Aug 20, 2020 at 9:18 AM Ed Smith <ph...@binaryflux.xyz> wrote:

> Hi,
>
> I needs info on few things before i start using ignite, my system is a big
> data , high eps (500k eps) , analytics system (gnn). I am right now
> inclined towards using native persistence.
>
> 1. Is it possible to create hot ,warm, cold and frozen data sets ?
> considering hot means most user data set and so on.
>
> 2. If i choose to use key value as storage , any api or language available
> to run queries over this data , specially aggregations (count, sum etc.)
>
> 3 Will i be able to create partitions based on some logical data types.
>
> 4. How do i setup a multi tenant architecture, specially where one user can
> have access to multiple tenants.
>
> 5. Is it possible to save data as tree in ignite ?
>
>
> I will appreciate if you could help me determine answers to these questions
> as i need to finalise on which framework to use in order to run above kind
> of system.
>
>
> Thanks
> ED.
>