You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@aurora.apache.org by "Bill Farner (JIRA)" <ji...@apache.org> on 2014/09/30 03:07:34 UTC
[jira] [Commented] (AURORA-759) Scheduler updater stuck in
ROLLING_BACK
[ https://issues.apache.org/jira/browse/AURORA-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152640#comment-14152640 ]
Bill Farner commented on AURORA-759:
------------------------------------
Ok, cool. I was able to reproduce. I believe this only happens then the job is originally stuck in {{THROTTLED}} or {{PENDING}}. I've turned up AURORA-769; and after patching that i'm unable to reproduce.
> Scheduler updater stuck in ROLLING_BACK
> ---------------------------------------
>
> Key: AURORA-759
> URL: https://issues.apache.org/jira/browse/AURORA-759
> Project: Aurora
> Issue Type: Bug
> Components: Scheduler
> Reporter: David McLaughlin
> Attachments: rollback-failure.log
>
>
> When using the new updater, I ran into a situation where my update was stuck in ROLLING_BACK state.
> My sequence of events:
> 1) Create a job where your package is being fetched from a local directory.
> 2) Have that local directory wiped clean.
> 3) Update your job with a new config, with a bad path.
> Stack traces from scheduler log:
> {code}
> I0926 18:50:56.888 THREAD278 org.apache.aurora.scheduler.updater.JobUpdateControllerImpl$4.execute: Forwarding task change for www-data/devel/hello_world/1
> E0926 18:50:56.888 THREAD278 org.apache.aurora.scheduler.updater.JobUpdateEventSubscriber.taskChangedState: Failed to handle state change: java.lang.IllegalStateException: Attempted
> to evaluate an inactive job updater.
> java.lang.IllegalStateException: Attempted to evaluate an inactive job updater.
> at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
> at org.apache.aurora.scheduler.updater.OneWayJobUpdater.evaluate(OneWayJobUpdater.java:145)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:451)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.access$700(JobUpdateControllerImpl.java:92)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl$4.execute(JobUpdateControllerImpl.java:239)
> at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult.apply(Storage.java:132)
> at org.apache.aurora.scheduler.storage.Storage$MutateWork$NoResult$Quiet.apply(Storage.java:149)
> at org.apache.aurora.scheduler.storage.log.LogStorage.write(LogStorage.java:551)
> at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.write(CallOrderEnforcingStorage.java:130)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.instanceChanged(JobUpdateControllerImpl.java:231)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.instanceChangedState(JobUpdateControllerImpl.java:216)
> at org.apache.aurora.scheduler.updater.JobUpdateEventSubscriber.taskChangedState(JobUpdateEventSubscriber.java:55)
> at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
> at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
> at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
> at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:304)
> at com.google.common.eventbus.EventBus.post(EventBus.java:275)
> at org.apache.aurora.scheduler.events.PubsubEventModule$2.post(PubsubEventModule.java:68)
> at org.apache.aurora.scheduler.state.StateManagerImpl.updateTaskAndExternalState(StateManagerImpl.java:443)
> at org.apache.aurora.scheduler.state.StateManagerImpl.access$100(StateManagerImpl.java:80)
> at org.apache.aurora.scheduler.state.StateManagerImpl$5.apply(StateManagerImpl.java:273)
> at org.apache.aurora.scheduler.state.StateManagerImpl$5.apply(StateManagerImpl.java:259)
> at org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:559)
> at org.apache.aurora.scheduler.storage.log.LogStorage$7.apply(LogStorage.java:556)
> at org.apache.aurora.scheduler.storage.mem.MemStorage.doWork(MemStorage.java:175)
> at org.apache.aurora.scheduler.storage.mem.MemStorage.access$400(MemStorage.java:59)
> at org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:200)
> at org.apache.aurora.scheduler.storage.mem.MemStorage$4.apply(MemStorage.java:197)
> {code}
> {code}
> java.lang.IllegalArgumentException: A terminal state should not specify actions: EvaluationResult{status=FAILED, sideEffects={4=SideEffect{action=Optional.absent(), statusChanges=[W
> ORKING, FAILED]}, 3=SideEffect{action=Optional.absent(), statusChanges=[WORKING, FAILED]}, 2=SideEffect{action=Optional.of(WATCH_TASK), statusChanges=[WORKING]}}}
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:490)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.changeJobUpdateStatus(JobUpdateControllerImpl.java:402)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.recordAndChangeJobUpdateStatus(JobUpdateControllerImpl.java:334)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.changeUpdateStatus(JobUpdateControllerImpl.java:319)
> at org.apache.aurora.scheduler.updater.JobUpdateControllerImpl.evaluateUpdater(JobUpdateControllerImpl.java:497)
> at org.apache.aurora.scheduler.storage.db.DbStorage.write(DbStorage.java:150)
> at org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497.CGLIB$write$4(<generated>)
> at org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497$$FastClassByGuice$$d23df519.invoke(<generated>)
> at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
> at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
> at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
> at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
> at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
> at org.apache.aurora.scheduler.storage.db.DbStorage$$EnhancerByGuice$$44df2497.write(<generated>)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)