[jira] [Commented] (AURORA-1615) Preemptor crashes scheduler during host maintenance

["Stephan Erb (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AURORA-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143186#comment-15143186 ] 

Stephan Erb commented on AURORA-1615:
-------------------------------------

Semi-related observation: Wouldn't it make sense to only index hosts without scheduled maintenance? Otherwise we might end up moving our precious production task to a host which we are about to drain rather soon. 

> Preemptor crashes scheduler during host maintenance
> ---------------------------------------------------
>
>                 Key: AURORA-1615
>                 URL: https://issues.apache.org/jira/browse/AURORA-1615
>             Project: Aurora
>          Issue Type: Bug
>          Components: Scheduler
>            Reporter: Maxim Khutornenko
>            Assignee: Maxim Khutornenko
>
> We have noticed an occasional scheduler failover when host maintenance is in effect:
> {noformat}
> To index multiple values under a key, use Multimaps.index.
>         at com.google.common.collect.Maps.uniqueIndex(Maps.java:1215)
>         at com.google.common.collect.Maps.uniqueIndex(Maps.java:1173)
>         at org.apache.aurora.scheduler.preemptor.PendingTaskProcessor.lambda$run$224(PendingTaskProcessor.java:130)
>         at org.apache.aurora.scheduler.storage.db.DbStorage.read(DbStorage.java:138)
>         at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
>         at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:83)
>         at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:570)
>         at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:113)
>         at org.apache.aurora.scheduler.preemptor.PendingTaskProcessor.run(PendingTaskProcessor.java:119)
> {noformat}
> Diffing colliding HostOffer objects revealed the only difference is in HostAttributes maintenance mode value: 
> mode=NONE vs. mode=DRAINING
> Upon examination it appears that it's quite possible to have duplicate HostOffer instances (same offer, same slave, different maintenance mode) due to the way [offers are accessed|https://github.com/apache/aurora/blob/9ed81a7db58f6a7cb308c8ac6a545705351c8c0e/src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java#L223-L226] as unmodifiable view over underlying ConcurrentSkipListSet. Here is the possible sequence:
> # Pending task processor starts [building unique index|https://github.com/apache/aurora/blob/2e2371481d9aaccd6a45ad0f442d963d5ae7a3c8/src/main/java/org/apache/aurora/scheduler/preemptor/PendingTaskProcessor.java#L128-L130] and the offers iterator pulls OfferA with mode=None
> # A host drain operation is initiated, a HostAttributesChanged event is raised
> # OfferManager [processes|https://github.com/apache/aurora/blob/9ed81a7db58f6a7cb308c8ac6a545705351c8c0e/src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java#L243-L246] HostAttributeChanged event and atomically [swaps|https://github.com/apache/aurora/blob/9ed81a7db58f6a7cb308c8ac6a545705351c8c0e/src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java#L315-L322] OfferA with OfferA' (mode=DRAINING)
> # iterator.next() inside of the uniqueIndex routine pulls OfferA' and the error is raised.
> We should either copy inside a synchronized getOffers() implementation or deal with possible duplicates at call site. I tend to think copying on access is a better approach. The only consumer of getOffers() is PendingTaskProcessor  with a relatively infrequent run loop (1 minute), so the perf impact of making a copy of all offers within a synchronized method should be acceptable. The alternative implies leaking the abstraction of host maintenance mode into the preemptor, which is less than ideal. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)