[jira] [Commented] (IGNITE-20503) Sql. Support big clusters by mapping service

["Andrey Mashenkov (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IGNITE-20503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17856710#comment-17856710 ] 

Andrey Mashenkov commented on IGNITE-20503:
-------------------------------------------

I'd suggest using more effective bitmap implementations like RoaringBitmap or Tree-encoded bitmap or else.

Typical query execution flow is running fragments on data nodes and fetch the data to a single node.
As I understand, mostly used execution targets will be OneOf (of a single node) and Partitioned (a bunch of very small node sets, comparing to cluster size). In most cases, I'd expect a BitMap will contains either a single 1-bit or have high clustering factor (i.e. how probable 1-bit will be followed by another 1-bit). In these cases, suggested implementations are very effective in both aspects: memory and performance. 
 

> Sql. Support big clusters by mapping service
> --------------------------------------------
>
>                 Key: IGNITE-20503
>                 URL: https://issues.apache.org/jira/browse/IGNITE-20503
>             Project: Ignite
>          Issue Type: Improvement
>          Components: sql
>            Reporter: Konstantin Orlov
>            Assignee: Andrey Mashenkov
>            Priority: Major
>              Labels: ignite-3
>             Fix For: 3.0.0-beta2
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> After refactoring made in IGNITE-20502, only clusters up to 64 nodes supported by MappingService.
> Let's implement one more ExecutionTargetFactory to support clusters of arbitrary size. See SmallClusterFactory and other classes in the same package for example.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)