[jira] [Created] (IGNITE-6019) SQL: client node should not hold the whole data set in-memory when possible

["Vladimir Ozerov (JIRA)" <ji...@apache.org>]

Vladimir Ozerov created IGNITE-6019:
---------------------------------------

             Summary: SQL: client node should not hold the whole data set in-memory when possible
                 Key: IGNITE-6019
                 URL: https://issues.apache.org/jira/browse/IGNITE-6019
             Project: Ignite
          Issue Type: Bug
          Components: sql
    Affects Versions: 2.1
            Reporter: Vladimir Ozerov
            Assignee: Alexander Paschenko
             Fix For: 2.2


Our SQL engine requests request data from server nodes in pieces called "page". This allows us to control memory consumption on client side. However, currently our client code is designed in a way that all pages are requested from all servers before a single cursor row is returned to the user. It defeats the whole idea of "cursor" and "page", and could easily crash client node with OOME. 

We need to fix that and request further pages in a kind of sliding window, keeping no more than "N" pages in memory simultaneously. Note that sometimes it is not possible, e.g. in case of {{DISTINCT}} or non-collocated {{GROUP BY}}. In this case we would have to build the whole result set first anyway. So let's focus on a scenario when the whole result set is not needed.

As currently everything is requested synchronously page-by-page, in the first version it would be enough to distribute synchronous page requests between cursor reads, without any prefetch. 

Implementation details:
1) Optimization should be applied only to {{skipMergeTbl=true}} cases, when complete result set of map queries is not needed.
2) Starting point is {{GridReduceQueryExecutor#query}}, see {{skipMergeTbl=true}} branch - this is where we get all pages eagerly.
3) Get no more than one page from the server at a time. We request the page, iterate over it, then request another page.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)