[jira] [Created] (HIVE-26937) Batch events during incremental replication to avoid O.O.M

["Rakshith C (Jira)" <ji...@apache.org>]

Rakshith C created HIVE-26937:
---------------------------------

             Summary: Batch events during incremental replication to avoid O.O.M
                 Key: HIVE-26937
                 URL: https://issues.apache.org/jira/browse/HIVE-26937
             Project: Hive
          Issue Type: Improvement
          Components: Hive
            Reporter: Rakshith C
            Assignee: Rakshith C


* Currently incremental replication flow of hive dumps all events read from notification logs sequentially in staging directory.
 * Repl Load loads all the event directories present in staging directory to a list and processes them.
 * This has caused O.O.M issues when number of events are large.

Hence introducing batching of events where Repl Dump dumps events in batches and Repl Load processes events batch by batch to avoid O.O.M.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)