You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oodt.apache.org by "Brian Foster (JIRA)" <ji...@apache.org> on 2010/12/09 20:20:00 UTC
[jira] Created: (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Ability to submit jobs to a DRMAA based resource manager
--------------------------------------------------------
Key: OODT-75
URL: https://issues.apache.org/jira/browse/OODT-75
Project: OODT
Issue Type: Improvement
Components: resource manager, workflow manager
Affects Versions: 0.1-incubating
Environment: none
Reporter: Brian Foster
Assignee: Brian Foster
Priority: Minor
Fix For: 0.2
Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
I believe this can be approached in two ways (or both):
- Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
- Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
The implementation of the Java DRMAA Specification has already been implemented:
- Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Updated] (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Chris A. Mattmann (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-75:
----------------------------------
- push out to 0.5
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.5
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Chris A. Mattmann (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-75:
----------------------------------
Fix Version/s: (was: 0.4)
0.5
- push to 0.5
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.5
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-75:
----------------------------------
Fix Version/s: (was: 0.3)
0.4
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.4
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970072#action_12970072 ]
Chris A. Mattmann commented on OODT-75:
---------------------------------------
++1, and great issue description!
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.2
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Paul Ramirez updated OODT-75:
-----------------------------
Fix Version/s: (was: 0.2)
0.3
Pushing off to 0.3 release.
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.3
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970380#action_12970380 ]
Chris A. Mattmann commented on OODT-75:
---------------------------------------
+1, this design looks great.
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.2
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (OODT-75) Ability to submit jobs to a DRMAA based
resource manager
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-75?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970342#action_12970342 ]
Brian Foster commented on OODT-75:
----------------------------------
- Here is the general idea in mind for wrapping DRMAA resource manager in CAS resource manager:
{noformat}
//This implementation spec is based on the idea the CAS-Resource will be used
// as a front to 1 or more DRMAA resource managers, thus each ResourceNode will
// be a DRMAA resource manager server. For example, if you have both Torque and
// Condor running, you would have two ResourceNodes (i.e. 1 for Torque and 1 for
// Condor).
public class DRMAAScheduler implements Scheduler {
public void run() {
//periodically getNextJob() from JobQueue
//pass the job to -> schedule(JobSpec)
}
public boolean schedule(JobSpec spec) throws SchedulerException {
//Get ResourceNodes supporting JobSpec queue
//Submit JobSpec to best ResourceNode
}
public ResourceNode nodeAvailable(JobSpec spec) throws SchedulerException {
//return null
}
public Monitor getMonitor() {
// return instance of DRMAAMonitor
}
public Batchmgr getBatchmgr() {
// return instance of DRMAABatchmgr
}
public JobQueue getJobQueue() {
//return instance of JobStack
}
public QueueManager getQueueManager() {
//return instance of QueueManager
}
}
public DRMAAMonitor implements Monitor {
public int getLoad(ResourceNode node) throws MonitorException {
//Ask DRMAA resource manager server at node.getIpAddr() for it current load
}
public List getNodes() throws MonitorException {
//Return a list all DRMAA resource managers
}
public ResourceNode getNodeById(String nodeId) throws MonitorException {
//Return a ResourceNode for a DRMAA resource manager for the given nodeId
}
public ResourceNode getNodeByURL(URL ipAddr) throws MonitorException {
//Return a ResourceNode for a DRMAA resource manager for the given ipAddr
}
public boolean reduceLoad(ResourceNode node, int loadValue) throws MonitorException {
//return true;
}
public boolean assignLoad(ResourceNode node, int loadValue)
throws MonitorException {
//return true;
}
}
public DRMAABatchmgr implements Batchmgr {
private Monitor monitor;
private JobRepository repository;
public boolean executeRemotely(JobSpec job, ResourceNode resNode)
throws JobExecutionException {
//send the JobSpec to the DRMAA resource manager at the given resNode.getIpAddr()
//update JobSpec in JobRepository
}
public void setMonitor(Monitor monitor) {
//this.monitor = monitor;
}
public void setJobRepository(JobRepository repository) {
//this.repository = repository;
}
public boolean killJob(String jobId, ResourceNode node) {
//tell the DRMAA resource manager at the given node.getIpAddr() to kill job with given jobId
}
public String getExecutionNode(String jobId) {
//get nodeId of DRMAA resource manager who is executing the job with given jobId
}
}
{noformat}
> Ability to submit jobs to a DRMAA based resource manager
> --------------------------------------------------------
>
> Key: OODT-75
> URL: https://issues.apache.org/jira/browse/OODT-75
> Project: OODT
> Issue Type: Improvement
> Components: resource manager, workflow manager
> Affects Versions: 0.1-incubating
> Environment: none
> Reporter: Brian Foster
> Assignee: Brian Foster
> Priority: Minor
> Fix For: 0.2
>
>
> Currently at JPL, the ACOS project is funding development to added the ability to submit CAS jobs to Torque PBS resource manager. In researching this topic, I found Torque was designed around DRMAA specification and serveral other resource management/grid components in use today also conform to DRMAA specification (e.g. Condor, SGE).
> I believe this can be approached in two ways (or both):
> - Write CAS-Resource plugins (e.g. Monitor, Batchmgr, etc...) such that CAS-Resource can wrap any resource manager component conforming to the Java DRMAA specification
> - Enable CAS-Workflow manager to submit jobs to a resource manager conforming to the Java DRMAA specification
> The implementation of the Java DRMAA Specification has already been implemented:
> - Gridway (http://www.gridway.org) has implement the Java DRMAA specification and is available for download under Apache licensing (http://www.gridway.org/doku.php?id=software), and when used with Globus which is also under Apache Public License (applicable for 4.0.1+) can be used to wrap DRMAA resource managers.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.