[jira] [Resolved] (HIVE-26937) Batch events during incremental replication to avoid O.O.M

["Rakshith C (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rakshith C resolved HIVE-26937.
-------------------------------
    Resolution: Fixed

> Batch events during incremental replication to avoid O.O.M
> ----------------------------------------------------------
>
>                 Key: HIVE-26937
>                 URL: https://issues.apache.org/jira/browse/HIVE-26937
>             Project: Hive
>          Issue Type: Improvement
>          Components: Hive
>            Reporter: Rakshith C
>            Assignee: Rakshith C
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> * Currently incremental replication flow of hive dumps all events read from notification logs sequentially in staging directory.
>  * Repl Load loads all the event directories present in staging directory to a list and processes them.
>  * This has caused O.O.M issues when number of events are large.
> Hence introducing batching of events where Repl Dump dumps events in batches and Repl Load processes events batch by batch to avoid O.O.M.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)