[jira] [Comment Edited] (SENTRY-2249) Persist HMS Full Snapshot in batches.

["kalyan kumar kalvagadda (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-2249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525670#comment-16525670 ] 

kalyan kumar kalvagadda edited comment on SENTRY-2249 at 6/27/18 10:14 PM:
---------------------------------------------------------------------------

*Here are two options I had in mind.*

*Option-1:* Persist the snapshot entities in batches. This may significantly reduce the DB operations. Currently there is one DB operation for one entry in snapshot. Which does not scale.

*Option-2:* Break the total snapshot into to batches and persist all of them in parallel in different transactions. As we use using repeatable_read isolation level we should be able to have parallel writes on the same table. This bring an issue if there is a failure in persisting any of the batches. This approach needs additional logic of cleaning the partially persisted snapshot.

 


was (Author: kkalyan):
*Here are two options I had in mind.*

*Option-1:* Persist the snapshot entities in batches. This may significantly reduce the DB operations. Currently there is one DB operation for one entry in snapshot. Which does not scale. 

*Option-2:* Break the total snapshot into to batches and persist all of them in parallel in different transactions. As we use using repeatable_read isolation level we should be able to have parallel writes on the same table. This bring an issue if there is a failure in persisting any of the batches. This approach needs additional logic of cleaning the partially persisted snapshot. I’m evaluating this option. 

 

> Persist HMS Full Snapshot in batches.
> -------------------------------------
>
>                 Key: SENTRY-2249
>                 URL: https://issues.apache.org/jira/browse/SENTRY-2249
>             Project: Sentry
>          Issue Type: Improvement
>          Components: Sentry
>    Affects Versions: 2.1.0
>            Reporter: kalyan kumar kalvagadda
>            Assignee: kalyan kumar kalvagadda
>            Priority: Major
>         Attachments: SENTRY-2249.001.patch
>
>
> Currently each entry in full snapshot of HMS is persisted one entry at a time. Instead it could be optimized by persisting the entries in batches. DB operations are expensive, reducing the number of database operations should help. This would decrease the time to persist the snapshot in to database significantly.
> Size of the batch could be configurable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)