You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Taner Ilyazov <ta...@gmail.com> on 2021/06/03 11:52:08 UTC

Re: Sorting result of ScanQuery

Okay, but since the nested object structure that I have contains a
Map<String, Object>, for which the idea is to have dynamic values, I'm not
sure how it will be handled. Do I need to create a separate table to do the
mapping of said Map<>? Couldn't find an example mapping a query entity
entry to a parameterized value.

On Wed, 26 May 2021 at 17:01, Stephen Darlington <
stephen.darlington@gridgain.com> wrote:

> A scan isn’t ordered. As you suspect, the way to order queries in Ignite
> is to use SQL.
>
> You don’t need to use annotations to define your SQL fields, indexes, etc.
> A slightly more verbose way is to use Query Entities (indexes
> <https://ignite.apache.org/docs/latest/SQL/indexes#configuring-indexes-using-query-entities>
> ).
>
> On 26 May 2021, at 14:24, Taner Ilyazov <ta...@gmail.com> wrote:
>
> Hello everyone,
>
> I'm new to the community and fairly new to Apache Ignite. I have a
> question for which I couldn't find a confirmation if it's possible or not.
>
> I have a use case where I need to persist a certain POJO to an ignite
> cluster. The POJO can not be changed, so adding @SqlQueryField to it's
> fields is not possible. Creating a data transfer object is an option, but I
> think adding mapping from/to the actual POJO will result in too much
> overhead, since performance requirements are really high.
>
> For now I'm using ScanQuery, but I could not find a way to sort the result
> based on a field value. So my main question is if it's possible and if not,
> what other options are there because the amount of data in question is too
> much for sorting on client side.
>
> If I take the SQL approach and introduce the mapping overhead between the
> DTO and POJO can I achieve server-side sorting on multiple nodes, keeping
> in mind that we'll have 1 table with a huge amount of data for writing and
> reading.
> Co-location if I understand correctly is ensuring all related data is on
> the same nodes, but in our case we have a single POJO which I would like
> it's data to be separated on different nodes for performance.
>
>
>
>
>
>

Re: Sorting result of ScanQuery

Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!

It sounds like you need some data normalization plus SQL secondary indexes.

Regards,
-- 
Ilya Kasnacheev


чт, 3 июн. 2021 г. в 17:11, Taner Ilyazov <ta...@gmail.com>:

> That's my concern. Because the requirements I have is persisting an Object
> with a complex nested structure, which can't be changed. I mean the class
> files can not be changed. Creating Data transfer objects and mapping
> between them is fine. But what we want to achieve is a really high write
> rate and the ability later to read data with filtering and sorting on the
> values of the hash map. I'm not sure how to achieve that without too much
> overhead.
>
> On Thu, 3 Jun 2021 at 15:29, Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> You can store collections in Ignite, the challenge is they’re effectively
>> invisible to SQL. In general it’s easiest to work with data in relational
>> structure. Ignite isn’t a document database.
>>
>> On 3 Jun 2021, at 12:52, Taner Ilyazov <ta...@gmail.com> wrote:
>>
>> Okay, but since the nested object structure that I have contains a
>> Map<String, Object>, for which the idea is to have dynamic values, I'm not
>> sure how it will be handled. Do I need to create a separate table to do the
>> mapping of said Map<>? Couldn't find an example mapping a query entity
>> entry to a parameterized value.
>>
>> On Wed, 26 May 2021 at 17:01, Stephen Darlington <
>> stephen.darlington@gridgain.com> wrote:
>>
>>> A scan isn’t ordered. As you suspect, the way to order queries in Ignite
>>> is to use SQL.
>>>
>>> You don’t need to use annotations to define your SQL fields, indexes,
>>> etc. A slightly more verbose way is to use Query Entities (indexes
>>> <https://ignite.apache.org/docs/latest/SQL/indexes#configuring-indexes-using-query-entities>
>>> ).
>>>
>>> On 26 May 2021, at 14:24, Taner Ilyazov <ta...@gmail.com> wrote:
>>>
>>> Hello everyone,
>>>
>>> I'm new to the community and fairly new to Apache Ignite. I have a
>>> question for which I couldn't find a confirmation if it's possible or not.
>>>
>>> I have a use case where I need to persist a certain POJO to an ignite
>>> cluster. The POJO can not be changed, so adding @SqlQueryField to it's
>>> fields is not possible. Creating a data transfer object is an option, but I
>>> think adding mapping from/to the actual POJO will result in too much
>>> overhead, since performance requirements are really high.
>>>
>>> For now I'm using ScanQuery, but I could not find a way to sort the
>>> result based on a field value. So my main question is if it's possible and
>>> if not, what other options are there because the amount of data in question
>>> is too much for sorting on client side.
>>>
>>> If I take the SQL approach and introduce the mapping overhead between
>>> the DTO and POJO can I achieve server-side sorting on multiple nodes,
>>> keeping in mind that we'll have 1 table with a huge amount of data for
>>> writing and reading.
>>> Co-location if I understand correctly is ensuring all related data is on
>>> the same nodes, but in our case we have a single POJO which I would like
>>> it's data to be separated on different nodes for performance.
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>

Re: Sorting result of ScanQuery

Posted by Taner Ilyazov <ta...@gmail.com>.
That's my concern. Because the requirements I have is persisting an Object
with a complex nested structure, which can't be changed. I mean the class
files can not be changed. Creating Data transfer objects and mapping
between them is fine. But what we want to achieve is a really high write
rate and the ability later to read data with filtering and sorting on the
values of the hash map. I'm not sure how to achieve that without too much
overhead.

On Thu, 3 Jun 2021 at 15:29, Stephen Darlington <
stephen.darlington@gridgain.com> wrote:

> You can store collections in Ignite, the challenge is they’re effectively
> invisible to SQL. In general it’s easiest to work with data in relational
> structure. Ignite isn’t a document database.
>
> On 3 Jun 2021, at 12:52, Taner Ilyazov <ta...@gmail.com> wrote:
>
> Okay, but since the nested object structure that I have contains a
> Map<String, Object>, for which the idea is to have dynamic values, I'm not
> sure how it will be handled. Do I need to create a separate table to do the
> mapping of said Map<>? Couldn't find an example mapping a query entity
> entry to a parameterized value.
>
> On Wed, 26 May 2021 at 17:01, Stephen Darlington <
> stephen.darlington@gridgain.com> wrote:
>
>> A scan isn’t ordered. As you suspect, the way to order queries in Ignite
>> is to use SQL.
>>
>> You don’t need to use annotations to define your SQL fields, indexes,
>> etc. A slightly more verbose way is to use Query Entities (indexes
>> <https://ignite.apache.org/docs/latest/SQL/indexes#configuring-indexes-using-query-entities>
>> ).
>>
>> On 26 May 2021, at 14:24, Taner Ilyazov <ta...@gmail.com> wrote:
>>
>> Hello everyone,
>>
>> I'm new to the community and fairly new to Apache Ignite. I have a
>> question for which I couldn't find a confirmation if it's possible or not.
>>
>> I have a use case where I need to persist a certain POJO to an ignite
>> cluster. The POJO can not be changed, so adding @SqlQueryField to it's
>> fields is not possible. Creating a data transfer object is an option, but I
>> think adding mapping from/to the actual POJO will result in too much
>> overhead, since performance requirements are really high.
>>
>> For now I'm using ScanQuery, but I could not find a way to sort the
>> result based on a field value. So my main question is if it's possible and
>> if not, what other options are there because the amount of data in question
>> is too much for sorting on client side.
>>
>> If I take the SQL approach and introduce the mapping overhead between the
>> DTO and POJO can I achieve server-side sorting on multiple nodes, keeping
>> in mind that we'll have 1 table with a huge amount of data for writing and
>> reading.
>> Co-location if I understand correctly is ensuring all related data is on
>> the same nodes, but in our case we have a single POJO which I would like
>> it's data to be separated on different nodes for performance.
>>
>>
>>
>>
>>
>>
>
>

Re: Sorting result of ScanQuery

Posted by Stephen Darlington <st...@gridgain.com>.
You can store collections in Ignite, the challenge is they’re effectively invisible to SQL. In general it’s easiest to work with data in relational structure. Ignite isn’t a document database. 

> On 3 Jun 2021, at 12:52, Taner Ilyazov <ta...@gmail.com> wrote:
> 
> Okay, but since the nested object structure that I have contains a Map<String, Object>, for which the idea is to have dynamic values, I'm not sure how it will be handled. Do I need to create a separate table to do the mapping of said Map<>? Couldn't find an example mapping a query entity entry to a parameterized value.
> 
> On Wed, 26 May 2021 at 17:01, Stephen Darlington <stephen.darlington@gridgain.com <ma...@gridgain.com>> wrote:
> A scan isn’t ordered. As you suspect, the way to order queries in Ignite is to use SQL.
> 
> You don’t need to use annotations to define your SQL fields, indexes, etc. A slightly more verbose way is to use Query Entities (indexes <https://ignite.apache.org/docs/latest/SQL/indexes#configuring-indexes-using-query-entities>).
> 
>> On 26 May 2021, at 14:24, Taner Ilyazov <taner.ilyazov@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hello everyone,
>> 
>> I'm new to the community and fairly new to Apache Ignite. I have a question for which I couldn't find a confirmation if it's possible or not.
>> 
>> I have a use case where I need to persist a certain POJO to an ignite cluster. The POJO can not be changed, so adding @SqlQueryField to it's fields is not possible. Creating a data transfer object is an option, but I think adding mapping from/to the actual POJO will result in too much overhead, since performance requirements are really high. 
>> 
>> For now I'm using ScanQuery, but I could not find a way to sort the result based on a field value. So my main question is if it's possible and if not, what other options are there because the amount of data in question is too much for sorting on client side.
>> 
>> If I take the SQL approach and introduce the mapping overhead between the DTO and POJO can I achieve server-side sorting on multiple nodes, keeping in mind that we'll have 1 table with a huge amount of data for writing and reading.
>> Co-location if I understand correctly is ensuring all related data is on the same nodes, but in our case we have a single POJO which I would like it's data to be separated on different nodes for performance.
>> 
>> 
>> 
> 
>