You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@aurora.apache.org by "Bill Farner (JIRA)" <ji...@apache.org> on 2014/01/25 05:09:37 UTC
[jira] [Commented] (AURORA-116) Improve efficiency of saving host
attributes (or avoid saving host attributes)
[ https://issues.apache.org/jira/browse/AURORA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881652#comment-13881652 ]
Bill Farner commented on AURORA-116:
------------------------------------
In a particularly-loaded cluster, we observed handling resourceOffers to take an inordinate amount of time:
{noformat}
I0125 04:06:06.030261 25455 sched.cpp:528] Scheduler::resourceOffers took 457.474384ms
I0125 04:06:07.431802 25456 sched.cpp:528] Scheduler::resourceOffers took 499.004174ms
I0125 04:06:08.715579 25452 sched.cpp:528] Scheduler::resourceOffers took 426.892948ms
I0125 04:06:10.422312 25458 sched.cpp:528] Scheduler::resourceOffers took 788.509018ms
I0125 04:06:11.531437 25450 sched.cpp:528] Scheduler::resourceOffers took 556.181547ms
I0125 04:06:12.849201 25452 sched.cpp:528] Scheduler::resourceOffers took 557.399593ms
I0125 04:06:14.131196 25446 sched.cpp:528] Scheduler::resourceOffers took 506.654534ms
I0125 04:06:15.558323 25457 sched.cpp:528] Scheduler::resourceOffers took 603.352069ms
I0125 04:06:16.797667 25454 sched.cpp:528] Scheduler::resourceOffers took 507.040296ms
I0125 04:06:18.342701 25449 sched.cpp:528] Scheduler::resourceOffers took 718.241925ms
I0125 04:06:22.795732 25445 sched.cpp:528] Scheduler::resourceOffers took 3.859263212secs
I0125 04:06:23.649204 25445 sched.cpp:528] Scheduler::resourceOffers took 838.23624ms
I0125 04:06:24.176681 25445 sched.cpp:528] Scheduler::resourceOffers took 522.324683ms
I0125 04:06:24.709750 25455 sched.cpp:528] Scheduler::resourceOffers took 328.162458ms
I0125 04:06:25.272554 25455 sched.cpp:528] Scheduler::resourceOffers took 559.136627ms
I0125 04:06:26.167621 25455 sched.cpp:528] Scheduler::resourceOffers took 875.709069ms
I0125 04:06:26.493263 25443 sched.cpp:528] Scheduler::resourceOffers took 134.088104ms
I0125 04:06:28.426606 25445 sched.cpp:528] Scheduler::resourceOffers took 420.597132ms
I0125 04:06:31.088995 25446 sched.cpp:528] Scheduler::resourceOffers took 1.262336563secs
I0125 04:06:31.934573 25456 sched.cpp:528] Scheduler::resourceOffers took 832.207275ms
I0125 04:06:33.181200 25454 sched.cpp:528] Scheduler::resourceOffers took 765.264834ms
I0125 04:06:35.013409 25457 sched.cpp:528] Scheduler::resourceOffers took 1.250660605secs
I0125 04:06:35.311099 25448 sched.cpp:528] Scheduler::resourceOffers took 230.108961ms
I0125 04:06:37.104035 25451 sched.cpp:528] Scheduler::resourceOffers took 624.894808ms
I0125 04:06:38.150378 25450 sched.cpp:528] Scheduler::resourceOffers took 427.445204ms
I0125 04:06:39.383716 25457 sched.cpp:528] Scheduler::resourceOffers took 392.365989ms
{noformat}
> Improve efficiency of saving host attributes (or avoid saving host attributes)
> ------------------------------------------------------------------------------
>
> Key: AURORA-116
> URL: https://issues.apache.org/jira/browse/AURORA-116
> Project: Aurora
> Issue Type: Task
> Components: Scheduler
> Reporter: Bill Farner
> Priority: Critical
>
> The scheduler performs multiple write operations for every resource offer, to save slave attributes:
> {noformat}
> public void resourceOffers(SchedulerDriver driver, List<Offer> offers) {
> Preconditions.checkState(registered, "Must be registered before receiving offers.");
> for (final Offer offer : offers) {
> log(Level.FINE, "Received offer: %s", offer);
> resourceOffers.incrementAndGet();
> storage.write(new MutateWork.NoResult.Quiet() {
> @Override protected void execute(MutableStoreProvider storeProvider) {
> storeProvider.getAttributeStore().saveHostAttributes(Conversions.getAttributes(offer));
> }
> });
> {noformat}
> This can unnecessarily block the singly-threaded message dispatch in the scheduler driver. An incremental improvement would be to aggregate all slave info and save it in one write operation. Better yet would be to perform writes asynchronously (taking care to not break task scheduling, since attributes are expected to be present). Even better yet, it would be great to determine if we can avoid storing host attributes.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)