[jira] [Commented] (AURORA-1023) Releasing the update lock trips off scheduler updater

["Maxim Khutornenko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14289729#comment-14289729 ] 

Maxim Khutornenko commented on AURORA-1023:
-------------------------------------------

Another nasty side-effect of having multiple active updates is failed quota checks:
{noformat}
 at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
        at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
        at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
        at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
        at com.google.common.collect.Maps.uniqueIndex(Maps.java:1166)
        at com.google.common.collect.Maps.uniqueIndex(Maps.java:1140)
        at com.google.common.collect.FluentIterable.uniqueIndex(FluentIterable.java:424)
        at org.apache.aurora.scheduler.quota.QuotaManager$QuotaManagerImpl$2.apply(QuotaManager.java:202)
        at org.apache.aurora.scheduler.quota.QuotaManager$QuotaManagerImpl$2.apply(QuotaManager.java:196)
        at org.apache.aurora.scheduler.storage.mem.MemStorage$2.apply(MemStorage.java:136)
        at org.apache.aurora.scheduler.storage.db.DbStorage.read(DbStorage.java:127)
        at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
        at org.apache.aurora.scheduler.storage.mem.MemStorage.read(MemStorage.java:133)
        at com.twitter.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:87)
        at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:646)
        at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:115)
        at org.apache.aurora.scheduler.quota.QuotaManager$QuotaManagerImpl.getQuotaInfo(QuotaManager.java:196)
        at org.apache.aurora.scheduler.quota.QuotaManager$QuotaManagerImpl.getQuotaInfo(QuotaManager.java:151)
        at org.apache.aurora.scheduler.thrift.SchedulerThriftInterface.getQuota(SchedulerThriftInterface.java:859)
{noformat}

We really need to enforce "one active update per job" invariant to avoid patching in random places. 

> Releasing the update lock trips off scheduler updater
> -----------------------------------------------------
>
>                 Key: AURORA-1023
>                 URL: https://issues.apache.org/jira/browse/AURORA-1023
>             Project: Aurora
>          Issue Type: Bug
>          Components: Scheduler
>            Reporter: Maxim Khutornenko
>            Assignee: Bill Farner
>            Priority: Critical
>
> Here is the faulty sequence:
> - User starts a scheduler job update and pauses while it's still in progress
> - User runs "aurora job cancel-update" command thus releasing the update lock
> - User starts a new scheduler job update
> At this point, any attempt to abort or pause an active update results in the following error [1]:
> {noformat}
> vagrant@vagrant-ubuntu-trusty-64:~$ aurora beta-update abort devcluster/www-data/prod/hello
>  INFO] Aborting update for: devcluster/www-data/prod/hello
> Failed to abort update due to error:
> 	expected one element but was: <JobUpdateSummary(updateId:4b7fdc14-428f-44e4-9261-908b606f47e2, jobKey:JobKey(role:www-data, environment:prod, name:hello), user:UNSECURE, state:JobUpdateState(status:ROLLING_FORWARD, createdTimestampMs:1421450382234, lastModifiedTimestampMs:1421450382234)), JobUpdateSummary(updateId:3c9c2fa2-8e51-4c13-8440-94364205a37b, jobKey:JobKey(role:www-data, environment:prod, name:hello), user:UNSECURE, state:JobUpdateState(status:ROLL_FORWARD_PAUSED, createdTimestampMs:1421450304935, lastModifiedTimestampMs:1421450324055))>
> {noformat}
> The only way to recover from this state is either wait for the active job update to reach terminal state or force it to it by running another cancel-update.
> While the "cancel-update" will eventually go away with the client updater, we do have a problem during the migration period. A possible (though ugly) short-term workaround could be calling "abortJobUpdate" from the "releaseLock" RPC.
> [1] - https://github.com/apache/incubator-aurora/blob/master/src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java#L295-L296



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)