You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@airavata.apache.org by Daniel Shchur <da...@huskers.unl.edu> on 2020/09/08 21:59:18 UTC
Submitting jobs on Open Science Grid
Hi all,
I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid.
We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated.
Thank you!
- Daniel Shchur
RE: Submitting jobs on Open Science Grid
Posted by Daniel Shchur <da...@huskers.unl.edu>.
Hi Dimuthu,
Thanks for the pointers, we really appreciate it!
-Daniel Shchur
-------- Original message --------
From: DImuthu Upeksha <di...@gmail.com>
Date: 9/16/20 15:54 (GMT-06:00)
To: Derek Weitzel <dw...@unl.edu>
Cc: Daniel Shchur <da...@huskers.unl.edu>, Airavata Users <us...@airavata.apache.org>
Subject: Re: Submitting jobs on Open Science Grid
Hi Derek,
That's great. I guess the best place I can refer for you to look is this [3]. That's how we keep track of the configurations of the job types. If you follow the trail, you should be able to implement a new HTCondorConfiguration implementation. Then we can write a groovy template like this [4] and attach to the configuration.
If you haven't already, you can set up Airavata in an IDE for development purposes. Please follow [5].
[3] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L107<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_config_JobFactory.java-23L107&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=-pAG6ofOPdzmyBRdh6CwlIi9aIGGQVLeivak0J2kfVM&e=>
[4] https://github.com/apache/airavata/blob/master/modules/configuration/server/src/main/resources/SLURM_Groovy.template<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_configuration_server_src_main_resources_SLURM-5FGroovy.template&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=Vz8i8NEkQqsHjuJnRaYk_XD1FIWPWKGkIN_GTuNI11o&e=>
[5] https://github.com/apache/airavata/tree/master/modules/ide-integration<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_tree_master_modules_ide-2Dintegration&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=GlSUcOFCGToWL_WCAoEvnhMkeaHl4cm2-4G-EvmeZkQ&e=>
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 3:26 PM Derek Weitzel <dw...@unl.edu>> wrote:
Hi,
For HTCondor, it is very similar submission process as SLURM. First, create the submission file describing the job. Then, submit that job using the command “condor_submit <job_file>”. It returns success or failure, as well as the job id (though, you will have to use some string parsing to extract the job id)
“condor_q <jobid>” is used to check the status. You can craft the command to only get a job status number. For example:
$ condor_q <jobid> -format ‘%i’ ‘JobStatus’
(or something like this) JobStatus numbers have very good definitions.
When the job is done, condor_q <jobid> will not return anything (it has left the queue). You can get completed job statistics with condor_history <jobid> -l
- Derek
> On Sep 16, 2020, at 12:55 PM, Daniel Shchur <da...@huskers.unl.edu>> wrote:
>
> Hi Dimuthu,
>
> I haven’t had a lot of experience with Condor. One of my colleagues, Derek, has more experience with this so I’ll have to ask him to help me a little. The way he explained it however is that there are equivalent commands to all of the SLURM commands. The submission script isn’t too different from what SLURM uses as well. This link should provide some insight: https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__htcondor.readthedocs.io_en_latest_users-2Dmanual_submitting-2Da-2Djob.html&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=ZcBQWH_IUK2pRw0VE2ceR56fMM_0nj0SD8tARdItYn4&e=>
>
> I’ll write up a step by step shortly.
>
> - Daniel Shchur
>
> From: DImuthu Upeksha
> Sent: Wednesday, September 16, 2020 1:45 PM
> To: Daniel Shchur
> Cc: Airavata Users
> Subject: Re: Submitting jobs on Open Science Grid
>
> Hi Daniel,
>
> Good thing about using a Groovy script and using cli to submit jobs is that you don't have to re-invent anything. You can simply follow what we have done for other schedulers and replicate them. I would also favor that approach than the python one. I just wasn't aware that HTCondor has that capability (CLI). Can you please summarize the steps (commands) that you normally follow to submit a job to HTCondor through the CLI?
>
> Thanks
> Dimuthu
>
> On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <da...@huskers.unl.edu>> wrote:
> Hi Dimuthu,
>
> Something we have been discussing is going the route of SLURM and implementing a Groovy template to submit jobs via CLI, as well as ping Condor to get status. Would it be preferable to try and go through python bindings instead to call the API directly?
>
> - Daniel Shchur
>
> From: DImuthu Upeksha
> Sent: Wednesday, September 16, 2020 1:30 PM
> To: Airavata Users
> Cc: Daniel Shchur
> Subject: Re: Submitting jobs on Open Science Grid
>
> Hi Daniel,
>
> If you follow the paper that Suresh has sent, you might notice that there is a Job submission task in the pre workflow manager. You have to implement a job submission task for HTCondor like we do for standard job schedulers like slurm and pbs. Have a look at [1] that implements a ssh based submission to schedulers. However, as I have heard, there is a python client for HTCondor submissions. So you don't need to worry about all these SSH based communications. You can use any of these [2] ways to talk to the python client through java.
>
> Implementing a job management extension for Airavata includes multiple steps so I would suggest you to work on that step by step. As the first step, try to come up with a sample java code that accepts all the parameters and submits a HTCondor job through the python client. Once you have a working code, I'll help you on how to generalize it and migrate it to Airavata.
>
> [1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=mm6zy3b2TnhIuJwi6Fdc7t2g-A4_YTqD9pVmbUib_c0&e=>
> [2] https://www.baeldung.com/java-working-with-python<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=heoYXejdfdGXsjC3OJadcZ28afz3ItjhxAGt9Jguku8&s=l9tCY2qN-UDP1ZtafSgyo0Y4Fk0tweVtWMzX-5LXJb4&e=>
>
> Thanks
> Dimuthu
>
> On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org>> wrote:
> Hi Daniel,
>
> As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework.
>
> Suresh
>
>
>
>
> On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>> wrote:
>
> Hi all,
>
> I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid.
>
> We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated.
>
> Thank you!
>
> - Daniel Shchur
Re: Submitting jobs on Open Science Grid
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Derek,
That's great. I guess the best place I can refer for you to look is this
[3]. That's how we keep track of the configurations of the job types. If
you follow the trail, you should be able to implement a new
HTCondorConfiguration implementation. Then we can write a groovy template
like this [4] and attach to the configuration.
If you haven't already, you can set up Airavata in an IDE for development
purposes. Please follow [5].
[3]
https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L107
[4]
https://github.com/apache/airavata/blob/master/modules/configuration/server/src/main/resources/SLURM_Groovy.template
[5] https://github.com/apache/airavata/tree/master/modules/ide-integration
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 3:26 PM Derek Weitzel <dw...@unl.edu> wrote:
> Hi,
>
> For HTCondor, it is very similar submission process as SLURM. First,
> create the submission file describing the job. Then, submit that job using
> the command “condor_submit <job_file>”. It returns success or failure, as
> well as the job id (though, you will have to use some string parsing to
> extract the job id)
>
> “condor_q <jobid>” is used to check the status. You can craft the command
> to only get a job status number. For example:
> $ condor_q <jobid> -format ‘%i’ ‘JobStatus’
> (or something like this) JobStatus numbers have very good definitions.
>
> When the job is done, condor_q <jobid> will not return anything (it has
> left the queue). You can get completed job statistics with condor_history
> <jobid> -l
>
>
>
> - Derek
>
>
>
> > On Sep 16, 2020, at 12:55 PM, Daniel Shchur <
> daniel.shchur@huskers.unl.edu> wrote:
> >
> > Hi Dimuthu,
> >
> > I haven’t had a lot of experience with Condor. One of my colleagues,
> Derek, has more experience with this so I’ll have to ask him to help me a
> little. The way he explained it however is that there are equivalent
> commands to all of the SLURM commands. The submission script isn’t too
> different from what SLURM uses as well. This link should provide some
> insight:
> https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html
> >
> > I’ll write up a step by step shortly.
> >
> > - Daniel Shchur
> >
> > From: DImuthu Upeksha
> > Sent: Wednesday, September 16, 2020 1:45 PM
> > To: Daniel Shchur
> > Cc: Airavata Users
> > Subject: Re: Submitting jobs on Open Science Grid
> >
> > Hi Daniel,
> >
> > Good thing about using a Groovy script and using cli to submit jobs is
> that you don't have to re-invent anything. You can simply follow what we
> have done for other schedulers and replicate them. I would also favor that
> approach than the python one. I just wasn't aware that HTCondor has that
> capability (CLI). Can you please summarize the steps (commands) that you
> normally follow to submit a job to HTCondor through the CLI?
> >
> > Thanks
> > Dimuthu
> >
> > On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <
> daniel.shchur@huskers.unl.edu> wrote:
> > Hi Dimuthu,
> >
> > Something we have been discussing is going the route of SLURM and
> implementing a Groovy template to submit jobs via CLI, as well as ping
> Condor to get status. Would it be preferable to try and go through python
> bindings instead to call the API directly?
> >
> > - Daniel Shchur
> >
> > From: DImuthu Upeksha
> > Sent: Wednesday, September 16, 2020 1:30 PM
> > To: Airavata Users
> > Cc: Daniel Shchur
> > Subject: Re: Submitting jobs on Open Science Grid
> >
> > Hi Daniel,
> >
> > If you follow the paper that Suresh has sent, you might notice that
> there is a Job submission task in the pre workflow manager. You have to
> implement a job submission task for HTCondor like we do for standard job
> schedulers like slurm and pbs. Have a look at [1] that implements a ssh
> based submission to schedulers. However, as I have heard, there is a python
> client for HTCondor submissions. So you don't need to worry about all these
> SSH based communications. You can use any of these [2] ways to talk to the
> python client through java.
> >
> > Implementing a job management extension for Airavata includes multiple
> steps so I would suggest you to work on that step by step. As the first
> step, try to come up with a sample java code that accepts all the
> parameters and submits a HTCondor job through the python client. Once you
> have a working code, I'll help you on how to generalize it and migrate it
> to Airavata.
> >
> > [1]
> https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java
> > [2] https://www.baeldung.com/java-working-with-python
> >
> > Thanks
> > Dimuthu
> >
> > On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org> wrote:
> > Hi Daniel,
> >
> > As we discussed offline, it will be good to get a contribution from you
> on the HTCondor extension. Please refer to this attached paper to get an
> idea about the job management framework.
> >
> > Suresh
> >
> >
> >
> >
> > On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>
> wrote:
> >
> > Hi all,
> >
> > I’m Daniel Shchur and working on a programming team at the University of
> Nebraska—Lincoln. We are working on creating a science gateway for a
> specific application that would allow users to run the application on UNL’s
> Holland Computing Center super computer as well as free resources on the
> Open Science Grid.
> >
> > We were wondering if it is at all possible to submit jobs to different
> job management workflows such as HTCondor, the workflow used by Open
> Science Grid. We were not able to find this in the documentation. Any help
> with this issue would be greatly appreciated.
> >
> > Thank you!
> >
> > - Daniel Shchur
>
>
RE: Submitting jobs on Open Science Grid
Posted by Daniel Shchur <da...@huskers.unl.edu>.
Hi Dimuthu,
I haven’t had a lot of experience with Condor. One of my colleagues, Derek, has more experience with this so I’ll have to ask him to help me a little. The way he explained it however is that there are equivalent commands to all of the SLURM commands. The submission script isn’t too different from what SLURM uses as well. This link should provide some insight: https://htcondor.readthedocs.io/en/latest/users-manual/submitting-a-job.html
I’ll write up a step by step shortly.
- Daniel Shchur
From: DImuthu Upeksha<ma...@gmail.com>
Sent: Wednesday, September 16, 2020 1:45 PM
To: Daniel Shchur<ma...@huskers.unl.edu>
Cc: Airavata Users<ma...@airavata.apache.org>
Subject: Re: Submitting jobs on Open Science Grid
Hi Daniel,
Good thing about using a Groovy script and using cli to submit jobs is that you don't have to re-invent anything. You can simply follow what we have done for other schedulers and replicate them. I would also favor that approach than the python one. I just wasn't aware that HTCondor has that capability (CLI). Can you please summarize the steps (commands) that you normally follow to submit a job to HTCondor through the CLI?
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <da...@huskers.unl.edu>> wrote:
Hi Dimuthu,
Something we have been discussing is going the route of SLURM and implementing a Groovy template to submit jobs via CLI, as well as ping Condor to get status. Would it be preferable to try and go through python bindings instead to call the API directly?
- Daniel Shchur
From: DImuthu Upeksha<ma...@gmail.com>
Sent: Wednesday, September 16, 2020 1:30 PM
To: Airavata Users<ma...@airavata.apache.org>
Cc: Daniel Shchur<ma...@huskers.unl.edu>
Subject: Re: Submitting jobs on Open Science Grid
Hi Daniel,
If you follow the paper that Suresh has sent, you might notice that there is a Job submission task in the pre workflow manager. You have to implement a job submission task for HTCondor like we do for standard job schedulers like slurm and pbs. Have a look at [1] that implements a ssh based submission to schedulers. However, as I have heard, there is a python client for HTCondor submissions. So you don't need to worry about all these SSH based communications. You can use any of these [2] ways to talk to the python client through java.
Implementing a job management extension for Airavata includes multiple steps so I would suggest you to work on that step by step. As the first step, try to come up with a sample java code that accepts all the parameters and submits a HTCondor job through the python client. Once you have a working code, I'll help you on how to generalize it and migrate it to Airavata.
[1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=>
[2] https://www.baeldung.com/java-working-with-python<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=>
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org>> wrote:
Hi Daniel,
As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework.
Suresh
On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>> wrote:
Hi all,
I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid.
We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated.
Thank you!
- Daniel Shchur
Re: Submitting jobs on Open Science Grid
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Daniel,
Good thing about using a Groovy script and using cli to submit jobs is that
you don't have to re-invent anything. You can simply follow what we have
done for other schedulers and replicate them. I would also favor that
approach than the python one. I just wasn't aware that HTCondor has that
capability (CLI). Can you please summarize the steps (commands) that you
normally follow to submit a job to HTCondor through the CLI?
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 2:37 PM Daniel Shchur <da...@huskers.unl.edu>
wrote:
> Hi Dimuthu,
>
>
>
> Something we have been discussing is going the route of SLURM and
> implementing a Groovy template to submit jobs via CLI, as well as ping
> Condor to get status. Would it be preferable to try and go through python
> bindings instead to call the API directly?
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha <di...@gmail.com>
> *Sent: *Wednesday, September 16, 2020 1:30 PM
> *To: *Airavata Users <us...@airavata.apache.org>
> *Cc: *Daniel Shchur <da...@huskers.unl.edu>
> *Subject: *Re: Submitting jobs on Open Science Grid
>
>
>
> Hi Daniel,
>
>
>
> If you follow the paper that Suresh has sent, you might notice that there
> is a Job submission task in the pre workflow manager. You have to implement
> a job submission task for HTCondor like we do for standard job schedulers
> like slurm and pbs. Have a look at [1] that implements a ssh based
> submission to schedulers. However, as I have heard, there is a python
> client for HTCondor submissions. So you don't need to worry about all these
> SSH based communications. You can use any of these [2] ways to talk to the
> python client through java.
>
>
>
> Implementing a job management extension for Airavata includes multiple
> steps so I would suggest you to work on that step by step. As the first
> step, try to come up with a sample java code that accepts all the
> parameters and submits a HTCondor job through the python client. Once you
> have a working code, I'll help you on how to generalize it and migrate it
> to Airavata.
>
>
>
> [1]
> https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=>
>
> [2] https://www.baeldung.com/java-working-with-python
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=>
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org> wrote:
>
> Hi Daniel,
>
>
>
> As we discussed offline, it will be good to get a contribution from you on
> the HTCondor extension. Please refer to this attached paper to get an idea
> about the job management framework.
>
>
>
> Suresh
>
>
>
>
>
>
>
> On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>
> wrote:
>
>
>
> Hi all,
>
>
>
> I’m Daniel Shchur and working on a programming team at the University of
> Nebraska—Lincoln. We are working on creating a science gateway for a
> specific application that would allow users to run the application on UNL’s
> Holland Computing Center super computer as well as free resources on the
> Open Science Grid.
>
> We were wondering if it is at all possible to submit jobs to different job
> management workflows such as HTCondor, the workflow used by Open Science
> Grid. We were not able to find this in the documentation. Any help with
> this issue would be greatly appreciated.
>
> Thank you!
>
>
>
> - Daniel Shchur
>
>
>
>
>
RE: Submitting jobs on Open Science Grid
Posted by Daniel Shchur <da...@huskers.unl.edu>.
Hi Dimuthu,
Something we have been discussing is going the route of SLURM and implementing a Groovy template to submit jobs via CLI, as well as ping Condor to get status. Would it be preferable to try and go through python bindings instead to call the API directly?
- Daniel Shchur
From: DImuthu Upeksha<ma...@gmail.com>
Sent: Wednesday, September 16, 2020 1:30 PM
To: Airavata Users<ma...@airavata.apache.org>
Cc: Daniel Shchur<ma...@huskers.unl.edu>
Subject: Re: Submitting jobs on Open Science Grid
Hi Daniel,
If you follow the paper that Suresh has sent, you might notice that there is a Job submission task in the pre workflow manager. You have to implement a job submission task for HTCondor like we do for standard job schedulers like slurm and pbs. Have a look at [1] that implements a ssh based submission to schedulers. However, as I have heard, there is a python client for HTCondor submissions. So you don't need to worry about all these SSH based communications. You can use any of these [2] ways to talk to the python client through java.
Implementing a job management extension for Airavata includes multiple steps so I would suggest you to work on that step by step. As the first step, try to come up with a sample java code that accepts all the parameters and submits a HTCondor job through the python client. Once you have a working code, I'll help you on how to generalize it and migrate it to Airavata.
[1] https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_airavata_blob_master_modules_airavata-2Dhelix_helix-2Dspectator_src_main_java_org_apache_airavata_helix_impl_task_submission_DefaultJobSubmissionTask.java&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=zN0maF1ewq6hAC6VMBZKPhZWRuwXE70h5w7A5NASKTc&e=>
[2] https://www.baeldung.com/java-working-with-python<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.baeldung.com_java-2Dworking-2Dwith-2Dpython&d=DwMFaQ&c=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo&r=SfMtyOx08LDuEVnPLH5bLf0tBX7T0FnnGw0oDSWu8Es&m=mcsIpz9s47afW619J62FW4ae7edw0b67402Whxf_Q_s&s=tJVmrIlkB6rca7GLYAle5AVZcIJY6Z6SkbajdJMEVgQ&e=>
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org>> wrote:
Hi Daniel,
As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework.
Suresh
On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>> wrote:
Hi all,
I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid.
We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated.
Thank you!
- Daniel Shchur
Re: Submitting jobs on Open Science Grid
Posted by DImuthu Upeksha <di...@gmail.com>.
Hi Daniel,
If you follow the paper that Suresh has sent, you might notice that there
is a Job submission task in the pre workflow manager. You have to implement
a job submission task for HTCondor like we do for standard job schedulers
like slurm and pbs. Have a look at [1] that implements a ssh based
submission to schedulers. However, as I have heard, there is a python
client for HTCondor submissions. So you don't need to worry about all these
SSH based communications. You can use any of these [2] ways to talk to the
python client through java.
Implementing a job management extension for Airavata includes multiple
steps so I would suggest you to work on that step by step. As the first
step, try to come up with a sample java code that accepts all the
parameters and submits a HTCondor job through the python client. Once you
have a working code, I'll help you on how to generalize it and migrate it
to Airavata.
[1]
https://github.com/apache/airavata/blob/master/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/DefaultJobSubmissionTask.java
[2] https://www.baeldung.com/java-working-with-python
Thanks
Dimuthu
On Wed, Sep 16, 2020 at 2:13 PM Suresh Marru <sm...@apache.org> wrote:
> Hi Daniel,
>
> As we discussed offline, it will be good to get a contribution from you on
> the HTCondor extension. Please refer to this attached paper to get an idea
> about the job management framework.
>
> Suresh
>
>
>
> On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu>
> wrote:
>
> Hi all,
>
> I’m Daniel Shchur and working on a programming team at the University of
> Nebraska—Lincoln. We are working on creating a science gateway for a
> specific application that would allow users to run the application on UNL’s
> Holland Computing Center super computer as well as free resources on the
> Open Science Grid.
>
> We were wondering if it is at all possible to submit jobs to different job
> management workflows such as HTCondor, the workflow used by Open Science
> Grid. We were not able to find this in the documentation. Any help with
> this issue would be greatly appreciated.
>
> Thank you!
>
> - Daniel Shchur
>
>
>
Re: Submitting jobs on Open Science Grid
Posted by Suresh Marru <sm...@apache.org>.
Hi Daniel,
As we discussed offline, it will be good to get a contribution from you on the HTCondor extension. Please refer to this attached paper to get an idea about the job management framework.
Suresh
> On Sep 8, 2020, at 5:59 PM, Daniel Shchur <da...@huskers.unl.edu> wrote:
>
> Hi all,
>
> I’m Daniel Shchur and working on a programming team at the University of Nebraska—Lincoln. We are working on creating a science gateway for a specific application that would allow users to run the application on UNL’s Holland Computing Center super computer as well as free resources on the Open Science Grid.
>
> We were wondering if it is at all possible to submit jobs to different job management workflows such as HTCondor, the workflow used by Open Science Grid. We were not able to find this in the documentation. Any help with this issue would be greatly appreciated.
>
> Thank you!
>
> - Daniel Shchur