[GitHub] drill pull request #928: DRILL-5716: Queue-driven memory allocation

[paul-rogers <gi...@git.apache.org>]

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/928#discussion_r136666594
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/rm/DefaultResourceManager.java ---
    @@ -0,0 +1,122 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + * http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.drill.exec.work.foreman.rm;
    +
    +import org.apache.drill.common.config.DrillConfig;
    +import org.apache.drill.exec.ops.QueryContext;
    +import org.apache.drill.exec.physical.PhysicalPlan;
    +import org.apache.drill.exec.server.BootStrapContext;
    +import org.apache.drill.exec.util.MemoryAllocationUtilities;
    +import org.apache.drill.exec.work.QueryWorkUnit;
    +import org.apache.drill.exec.work.foreman.Foreman;
    +
    +/**
    + * Represents a default resource manager for clusters that do not provide query
    + * queues. Without queues to provide a hard limit on the query admission rate,
    + * the number of active queries must be estimated and the resulting resource
    + * allocations will be rough estimates.
    + */
    +
    +public class DefaultResourceManager implements ResourceManager {
    +
    +  public static class DefaultResourceAllocator implements QueryResourceAllocator {
    +
    +    private QueryContext queryContext;
    +
    +    protected DefaultResourceAllocator(QueryContext queryContext) {
    +      this.queryContext = queryContext;
    +    }
    +
    +    @Override
    +    public void visitAbstractPlan(PhysicalPlan plan) {
    +      boolean replanMemory = ! plan.getProperties().hasResourcePlan;
    +      if (! replanMemory || plan == null) {
    +        return;
    +      }
    +      MemoryAllocationUtilities.setupBufferedOpsMemoryAllocations(plan, queryContext);
    +    }
    +
    +    @Override
    +    public void visitPhysicalPlan(QueryWorkUnit work) {
    +    }
    +  }
    +
    +  public static class DefaultQueryResourceManager extends DefaultResourceAllocator implements QueryResourceManager {
    +
    +    @SuppressWarnings("unused")
    +    private final DefaultResourceManager rm;
    +
    +    public DefaultQueryResourceManager(final DefaultResourceManager rm, final Foreman foreman) {
    +      super(foreman.getQueryContext());
    +      this.rm = rm;
    +    }
    +
    +    @Override
    +    public void setCost(double cost) {
    +      // Nothing to do. The EXECUTION option in Foreman calls this,
    +      // but does not do the work to plan sort memory. Is EXECUTION
    +      // even used?
    +    }
    +
    +    @Override
    +    public void admit() {
    +      // No queueing by default
    +    }
    +
    +    @Override
    +    public void exit() {
    +      // No queueing by default
    +    }
    +
    +    @Override
    +    public boolean hasQueue() { return false; }
    +
    +    @Override
    +    public String queueName() { return null; }
    +  }
    +
    +  BootStrapContext bootStrapContext;
    +  public long memoryPerNode;
    +  public int cpusPerNode;
    +
    +  public DefaultResourceManager() {
    +    memoryPerNode = DrillConfig.getMaxDirectMemory();
    +    cpusPerNode = Runtime.getRuntime().availableProcessors();
    --- End diff --
    
    This just grabs the total actual CPUs. Code elsewhere would do calcs based on the number. Same way that the code here grabs total memory, but other code does calcs on that (such as setting aside a reserve.)
    
    But, CPU allocation is not yet implemented so this code is a bit of a dead end at the moment. However, did not want to remove it 'cause I'd have to figure it out again later when we add CPU allocation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---