[jira] [Created] (IGNITE-18033) Implement cooperative GC of MV data during RAFT commands execution

["Ivan Bessonov (Jira)" <ji...@apache.org>]

Ivan Bessonov created IGNITE-18033:
--------------------------------------

             Summary: Implement cooperative GC of MV data during RAFT commands execution
                 Key: IGNITE-18033
                 URL: https://issues.apache.org/jira/browse/IGNITE-18033
             Project: Ignite
          Issue Type: Improvement
            Reporter: Ivan Bessonov


Please refer to the Epic for the basic description. Please also refer to IGNITE-18031 for naive implementation details and general thoughts.

Technically, there is a possibility that the background GC process wouldn't catch up if there's too much data being loaded into the system. Scanning through the entire partition takes time, and only a small subset of data could be under a constant stream of modification.

To account for that, each update can be preceded with the manual GC of that row. In this case, there's less work for the background processor, and there's an empirical sense that frequently updated data will be just as frequently vacuumed, thus not allowing too much garbage to appear in the first place.

Now that's not a guarantee that there would be no problems at all, so we should also think about other ways of cooperative GC.

Of course, it would be nice to have a queue of "old" rows ready for you to clean. How feasible is it? It would have to be supported by every engine. That's extra data and extra time for management. Looks pretty much exactly like a TTL manager in Ignite 2.x. I assume that we don't implement anything like that right now, starting with more naive approaches. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)