Re: Pre-calculated JOIN queries as map in ignite

[vkulichenko <va...@gmail.com>]

Hi Dharam,

Can you please properly subscribe to the mailing list so that the community
can receive email notifications? Here is the instruction:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


dharamthacker wrote
> I am new to ignite and POCing currently. I have a question regarding ways
> to store/load data in map. It's bit tricky and strange requirement. 
*
> Example:
*

> I have Employee, Department, Project [Tables in database] + [Entity
> classes in application]. But I don't want to store each of these in a
> separate map in memory but rather I want to store pre-calculated join
> results in a designated map.
*
> Dynamic Query : 
*
> select
> employeeId,employeeName,departmentName,projectName,projectStart,projectEnd
> from Employee,Department,Project where $JOIN
> 
> I know at least before hand that, what would be key fields and what would
> be value fields. From above example, I can denote my "Map" as shown below,
> 
> (Key=Set
> <Tuple>
> , Value = List
> <Tuple>
> )
> 
> Key : Set (employeeId,departmentId) , Value : List
> [(employeeName,value),(departmentName,value),(projectName,value),(projectStart,value),(projectEnd,value)]
> 
> So you can see with every pair of (employeeId,departmentId) I would be
> having multiple values associates with it. But dilemma is I don't have
> domain model/entity pojos before hand. Such dynamic views/maps can be
> added flexibly so that we don't have to go and change domain/entity model
> every time. We don't want to do joins/calculations every time for
> thousands of such client request on every call.
> 
> Is it possible to fire such join queries using MapLoader or by any other
> means? I can think of Map with (Key=Set
> <Tuple>
> , Value = List
> <Tuple>
> ) as data structure to store final results.Any other better alternative?
> 
> Could there be any performance issues while retrieving values from such
> map based on keys? Any memory optimizations I should take care of?
> 
> Thanks, Dharam

You are not required to use SQL queries. It's fine to use Ignite as a simple
caching mechanism for DB query results. Each time a query is executed, save
the result in IgniteCache and then use this cached result is the same query
is requested. You can also use expirations [1] and/or evictions [2] to make
sure that you don't have too much data in the cache and don't run out of
memory.

[1] https://apacheignite.readme.io/docs/expiry-policies
[2] https://apacheignite.readme.io/docs/evictions

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Pre-calculated-JOIN-queries-as-map-in-ignite-tp5294p5309.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Pre-calculated JOIN queries as map in ignite

[vkulichenko <va...@gmail.com>]

dharamthacker wrote
> But that's what I want to avoid [Each time DB query].

I meant that you go to the DB only if the data is not in the cache (was not
cached yet or was already evicted). If it's already there, you just use
IgniteCache.get().

dharamthacker wrote
> But I don't want to have predefined POJO to store it into cache. I should
> be able to add dynamic query each resulting into separate map in
> distributed cache.

Ignite stores all the data in binary format which already supports dynamic
schema changes. It also provides the builder for binary objects which you
can use in case you don't have classes. See this page for details:
https://apacheignite.readme.io/docs/binary-marshaller

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Pre-calculated-JOIN-queries-as-map-in-ignite-tp5294p5325.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Pre-calculated JOIN queries as map in ignite

[dharamthacker <dh...@gmail.com>]

Hi,

As you mentioned,
"
It's fine to use Ignite as a simple caching mechanism for DB query results.
Each time a query is executed, save the result in IgniteCache and then use
this cached result is the same query is requested
"

But that's what I want to avoid [Each time DB query]. On startup itself, I
want to cache all JOINed resultset into memory as I am sure that it's going
to be required. But I don't have POJO/domain model, then how I should store
them in cache?

If I say Employee or Department, they denote domain/entity model classes.
But I don't want to have predefined POJO to store it into cache. I should be
able to add dynamic query each resulting into separate map in distributed
cache. That's a reason, I mentioned,
Key = Set<Tuple>
Value = List<Tuple>

I don't want to store database tables in cache and then apply
predicate/query to calculate join. Rather I only want to have precalculated
JOINed results. 

- User1 may say, given A,B I need A1,B1,B2,B3,B4,B5 [Separate Map with
Key(A,B) Values(A1,B1,B2,B3,B4,B5)]
- User2 may say, given A,C I need A1,C1,C5 [Separate Map with Key(A,C)
Values (A1,C1,C5)]

Thanks,
Dharam



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Pre-calculated-JOIN-queries-as-map-in-ignite-tp5294p5314.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.