[jira] [Commented] (SENTRY-2185) Performance Issue: Saving MAuthzPathsMapping should be done in batch for path full snapshot

["Na Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16407260#comment-16407260 ] 

Na Li commented on SENTRY-2185:
-------------------------------

h3. The following may be useful to persist path full snapshot in bulk in http://www.datanucleus.org/products/accessplatform_3_2/jdo/performance_tuning.html
{code:java}
Persistence Process
To optimise the persistence process for performance you need to analyse what operations are performed and when, to see if there are some features that you could disable to get the persistence you require and omit what is not required. If you think of a typical transaction, the following describes the process

Start the transaction (if running non-transactional then this is seamless)
Perform persistence operations.
If you are using "optimistic" transactions then all datastore operations will be delayed until commit. Otherwise all datastore operations will default to being performed immediately. If you are handling a very large number of objects in the transaction you would benefit by either disabling "optimistic" transactions, or alternatively setting the persistence property datanucleus.flush.mode to AUTO, or alternatively again do a manual flush every "n" objects, like this
for (int i=0;i<1000000;i++) { if ((i%10000)/10000 == 0 && i != 0) { pm.flush(); } ... }
If you are retrieving any object by its identity (pm.getObjectById))and know that it will be present in the Level2 cache, for example, you can set the persistence property datanucleus.findObject.validateWhenCached to false and this will skip a separate call to the datastore to validate that the object exists in the datastore.{code}
h3.  

> Performance Issue: Saving MAuthzPathsMapping should be done in batch for path full snapshot
> -------------------------------------------------------------------------------------------
>
>                 Key: SENTRY-2185
>                 URL: https://issues.apache.org/jira/browse/SENTRY-2185
>             Project: Sentry
>          Issue Type: Bug
>          Components: Sentry
>    Affects Versions: 2.1.0
>            Reporter: Na Li
>            Priority: Major
>
> From the code below, for each MAuthzPathsMapping, there is a query to save each MAuthzPathsMapping instance. Need to save the changes in batch. 
> {code:java}
>   public void persistFullPathsImage(final Map<String, Collection<String>> authzPaths,
>       final long notificationID) throws Exception {
>     tm.executeTransactionWithRetry(
>             pm -> {
>               pm.setDetachAllOnCommit(false); // No need to detach objects
>               deleteNotificationsSince(pm, notificationID + 1);
>               // persist the notidicationID
>               pm.makePersistent(new MSentryHmsNotification(notificationID));
>               // persist the full snapshot
>               long snapshotID = getCurrentAuthzPathsSnapshotID(pm);
>               long nextSnapshotID = snapshotID + 1;
>               pm.makePersistent(new MAuthzPathsSnapshotId(nextSnapshotID));
>               LOGGER.info("Attempting to commit new HMS snapshot with ID = {}", nextSnapshotID);
>               for (Map.Entry<String, Collection<String>> authzPath : authzPaths.entrySet()) {
>                 pm.makePersistent(new MAuthzPathsMapping(nextSnapshotID, authzPath.getKey(), authzPath.getValue()));
>               }
>               return null;
>             });
>   }{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)