[jira] [Created] (STORM-55) Replace Storm's scheduler with a constraint logic programming engine

["James Xu (JIRA)" <ji...@apache.org>]

James Xu created STORM-55:
-----------------------------

             Summary: Replace Storm's scheduler with a constraint logic programming engine
                 Key: STORM-55
                 URL: https://issues.apache.org/jira/browse/STORM-55
             Project: Apache Storm (Incubating)
          Issue Type: New Feature
            Reporter: James Xu


https://github.com/nathanmarz/storm/issues/383

CLP seems to be a great fit for Storm's resource scheduling. We want to be able to declaratively specify constraints, such as:

1. Topology A's slots should be <= 10 and as close to 10 as possible (minimize the delta between assigned slots and 10)
2. All topologies should use less than 200 CPU's and less than 600 GB of memory
3. Topology B should run at most 2 workers on each host
4. Each worker for topology C should run at most one task for component X and one task for component Y
5. Should minimize the amount of reassignment to running topologies in order to satisfy constraints
6. Should only be allowed to reassign workers for an individual topology whose individual constraints are satisfied once every 10 minutes

And then the logic engine should do an efficient search and optimize the constraints.

What's going to make this especially powerful is exposing the logic engine to users (so people can plug in functions that return new constraints), making it really easy for people to add sophisticated scheduling logic with minimal effort.

Clojure's core.logic may be a great fit for this. According to this thread it doesn't do minimization goals ( https://groups.google.com/group/clojure/browse_thread/thread/a050a4e6b390514a ), but it may in the future.

-------------------------------------------
@nathanmarz: There also appears to be a fair amount of research in this area that we should look at: http://scholar.google.com/scholar?q=resource+scheduling+constraint+logic+programming&hl=en&as_sdt=0&as_vis=1&oi=scholart&sa=X&ei=X1uIUK6gA4rBigL5yoCgBg&ved=0CD0QgQMwAA


-------------------------------------------
bgedik: It would be good if the default scheduler understands placement constraints. Would be very helpful if the placement constraints can be attached to the topology, relieving the developer from having to write a scheduler to get the placement she wants.

The placement constraints can come in various forms. Here are some examples:

Isolation constraint: The host/spout task cannot share its host with any other task
Co-location constraint: A group of host/spout tasks has to be placed on the same host
Ex-location constraint: A group of host/spout tasks cannot be placed on the same host
Pool placement: A group of host/spout tasks are sprayed over a pool of hosts
Explicit placement: A host/spout task is placed on a user-specified host
Pools can be defined using host properties. For instance, configuration files can be used to assign properties to hosts. As a specific example, some hosts can have GPUs installed. As an application developer, I can request 4 such hosts (with gpu=true property) to create a pool in my topology construction code (without hardcoding the identities of the hosts). Then I can spray some of my bolt instances across the hosts in my GPU pool.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)