[jira] [Updated] (AURORA-1861) Remove duplicate Snapshot fields for DB stores

["David McLaughlin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AURORA-1861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David McLaughlin updated AURORA-1861:
-------------------------------------
    Description: 
Currently we double-write any DB-backed stores into a Snapshot struct when creating a Snapshot. This inflates the size of the Snapshot, which is already a problem for large production clusters (see AURORA-74). 

Example for LockStore from https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java:
{code}
      new SnapshotField() {
        // It's important for locks to be replayed first, since there are relations that expect
        // references to be valid on insertion.
        @Override
        public void saveToSnapshot(MutableStoreProvider store, Snapshot snapshot) {
          snapshot.setLocks(ILock.toBuildersSet(store.getLockStore().fetchLocks()));
        }

        @Override
        public void restoreFromSnapshot(MutableStoreProvider store, Snapshot snapshot) {
          if (hasDbSnapshot(snapshot)) {
            LOG.info("Deferring lock restore to dbsnapshot");
            return;
          }

          store.getLockStore().deleteLocks();

          if (snapshot.isSetLocks()) {
            for (Lock lock : snapshot.getLocks()) {
              store.getLockStore().saveLock(ILock.build(lock));
            }
          }
        }
      },
{code}

The saveToSnapshot here is totally redundant as the entire H2 database is dumped into the dbScript field. 

Note: one major side-effect here is if anyone is trying to read these snapshots and utilize the data outside of Java - they'll lose the ability to process the data without being able to apply the DB script. 

  was:
Currently we double-write any DB-backed stores into a Snapshot struct when creating a Snapshot. This inflates the size of the Snapshot, which is already a problem for large production clusters (see AURORA-74). 

Example for LockStore from https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java:
{code}
      new SnapshotField() {
        // It's important for locks to be replayed first, since there are relations that expect
        // references to be valid on insertion.
        @Override
        public void saveToSnapshot(MutableStoreProvider store, Snapshot snapshot) {
          snapshot.setLocks(ILock.toBuildersSet(store.getLockStore().fetchLocks()));
        }

        @Override
        public void restoreFromSnapshot(MutableStoreProvider store, Snapshot snapshot) {
          if (hasDbSnapshot(snapshot)) {
            LOG.info("Deferring lock restore to dbsnapshot");
            return;
          }

          store.getLockStore().deleteLocks();

          if (snapshot.isSetLocks()) {
            for (Lock lock : snapshot.getLocks()) {
              store.getLockStore().saveLock(ILock.build(lock));
            }
          }
        }
      },
{code}

Note: one major side-effect here is if anyone is trying to read these snapshots and utilize the data outside of Java - they'll lose the ability to process the data without being able to apply the DB script. 


> Remove duplicate Snapshot fields for DB stores
> ----------------------------------------------
>
>                 Key: AURORA-1861
>                 URL: https://issues.apache.org/jira/browse/AURORA-1861
>             Project: Aurora
>          Issue Type: Task
>          Components: Scheduler
>            Reporter: David McLaughlin
>
> Currently we double-write any DB-backed stores into a Snapshot struct when creating a Snapshot. This inflates the size of the Snapshot, which is already a problem for large production clusters (see AURORA-74). 
> Example for LockStore from https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImpl.java:
> {code}
>       new SnapshotField() {
>         // It's important for locks to be replayed first, since there are relations that expect
>         // references to be valid on insertion.
>         @Override
>         public void saveToSnapshot(MutableStoreProvider store, Snapshot snapshot) {
>           snapshot.setLocks(ILock.toBuildersSet(store.getLockStore().fetchLocks()));
>         }
>         @Override
>         public void restoreFromSnapshot(MutableStoreProvider store, Snapshot snapshot) {
>           if (hasDbSnapshot(snapshot)) {
>             LOG.info("Deferring lock restore to dbsnapshot");
>             return;
>           }
>           store.getLockStore().deleteLocks();
>           if (snapshot.isSetLocks()) {
>             for (Lock lock : snapshot.getLocks()) {
>               store.getLockStore().saveLock(ILock.build(lock));
>             }
>           }
>         }
>       },
> {code}
> The saveToSnapshot here is totally redundant as the entire H2 database is dumped into the dbScript field. 
> Note: one major side-effect here is if anyone is trying to read these snapshots and utilize the data outside of Java - they'll lose the ability to process the data without being able to apply the DB script. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)