You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Eric Kuyt <er...@wur.nl> on 2016/02/24 15:03:30 UTC
Adding a SGE resourceJobManagerType
Hey all, I am playing around with airavata and when I can run jobs on my
SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
the Executing state.
Now I try to make my own SGEJobConfiguration class.
Created the class plus outputParser in org.apache.airavata.gfac.impl.job
- add SGE
in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
- add SGE
in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
- add SGE to
airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
What should the next step be? if I Wanted to create my own SGEOutputparser?
PGA now gives me a Required field 'resourceJobManagerType' is unset! when I
select SGE in the list.
Thanks,
Eric
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl
Web: http://www.cvi.wur.nl
Re: Adding a SGE resourceJobManagerType
Posted by "Pierce, Marlon" <ma...@iu.edu>.
+1 for asking more questions if you hit a problem. We would also be interested in receiving this as a contribution to Airavata.
Marlon
From: Shameera Rathnayaka <sh...@gmail.com>>
Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Date: Wednesday, February 24, 2016 at 10:17 AM
To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Subject: Re: Adding a SGE resourceJobManagerType
Hi Eric,
There are couple of things you need to do to add a new jobmanager type.
1. Configure your SGE resource type in gfac yaml config file (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config). As you can see you need to provide an output parser and email parser for your SGE, Unless you can't use existing implementations you need to come up with your own implementations of these interfaces.
2. Don't modify thrift generated file, instead add your new ResourceJobManagerType to airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift file and generate thrift classes.
3. Come up with SGE JobConfiguration class, and modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to provide it. You should have xslt template for your SGE script file.
let us know if you hit any issue while you following these steps.
Thanks,
Shameera.
On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <ka...@gmail.com>> wrote:
Hi Eric,
To understand what you did, you added a new ResourceJobManagerType in the thrift definition as "SGE" ? I assume you generate thrift stubs after adding it. For that, you need to have thrift 0.93 installed in your system.
Thanks..
Chathuri
On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl>> wrote:
Hey all, I am playing around with airavata and when I can run jobs on my SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave the Executing state.
Now I try to make my own SGEJobConfiguration class.
Created the class plus outputParser in org.apache.airavata.gfac.impl.job
- add SGE in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
- add SGE in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
- add SGE to airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
What should the next step be? if I Wanted to create my own SGEOutputparser?
PGA now gives me a Required field 'resourceJobManagerType' is unset! when I select SGE in the list.
Thanks,
Eric
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
Web: http://www.cvi.wur.nl
--
Shameera Rathnayaka
Re: Adding a SGE resourceJobManagerType
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Hi Eric, did this work for you?
From: Suresh Marru <sm...@apache.org>>
Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Date: Wednesday, February 24, 2016 at 10:30 AM
To: Airavata Dev <de...@airavata.apache.org>>
Subject: Re: Adding a SGE resourceJobManagerType
SGE - Sun Grid Engine is the old variant of Univa Grid Engine, so you should pick UGE in Airavata and should be good to go. SGE/UGE implement PBS protocol so parsing should be similar. Lonestar4 used to have UGE, so email parsing should be similar to that.
Suresh
On Feb 24, 2016, at 10:17 AM, Shameera Rathnayaka <sh...@gmail.com>> wrote:
Hi Eric,
There are couple of things you need to do to add a new jobmanager type.
1. Configure your SGE resource type in gfac yaml config file (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config). As you can see you need to provide an output parser and email parser for your SGE, Unless you can't use existing implementations you need to come up with your own implementations of these interfaces.
2. Don't modify thrift generated file, instead add your new ResourceJobManagerType to airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift file and generate thrift classes.
3. Come up with SGE JobConfiguration class, and modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to provide it. You should have xslt template for your SGE script file.
let us know if you hit any issue while you following these steps.
Thanks,
Shameera.
On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <ka...@gmail.com>> wrote:
Hi Eric,
To understand what you did, you added a new ResourceJobManagerType in the thrift definition as "SGE" ? I assume you generate thrift stubs after adding it. For that, you need to have thrift 0.93 installed in your system.
Thanks..
Chathuri
On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl>> wrote:
Hey all, I am playing around with airavata and when I can run jobs on my SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave the Executing state.
Now I try to make my own SGEJobConfiguration class.
Created the class plus outputParser in org.apache.airavata.gfac.impl.job
- add SGE in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
- add SGE in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
- add SGE to airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
What should the next step be? if I Wanted to create my own SGEOutputparser?
PGA now gives me a Required field 'resourceJobManagerType' is unset! when I select SGE in the list.
Thanks,
Eric
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
Web: http://www.cvi.wur.nl<http://www.cvi.wur.nl/>
--
Shameera Rathnayaka
Re: Adding a SGE resourceJobManagerType
Posted by Eric Kuyt <er...@wur.nl>.
Hey everyone, thanks for all the enthusiastic reactions, I think I can play
around with this tomorrow.
I noticed earlier however that the pbs file generated for univa grid engine
was giving an error code on qsub. Will read in univa today and if it is
supposed tot be backwards compatible.
Eric
On 24 February 2016 at 22:15, Pierce, Marlon <ma...@iu.edu> wrote:
> Hi Eric, did this work for you?
>
> From: Suresh Marru <sm...@apache.org>>
> Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <
> dev@airavata.apache.org<ma...@airavata.apache.org>>
> Date: Wednesday, February 24, 2016 at 10:30 AM
> To: Airavata Dev <de...@airavata.apache.org>>
> Subject: Re: Adding a SGE resourceJobManagerType
>
> SGE - Sun Grid Engine is the old variant of Univa Grid Engine, so you
> should pick UGE in Airavata and should be good to go. SGE/UGE implement PBS
> protocol so parsing should be similar. Lonestar4 used to have UGE, so email
> parsing should be similar to that.
>
> Suresh
>
>
> On Feb 24, 2016, at 10:17 AM, Shameera Rathnayaka <shameerainfo@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Eric,
>
> There are couple of things you need to do to add a new jobmanager type.
>
> 1. Configure your SGE resource type in gfac yaml config file
> (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config).
> As you can see you need to provide an output parser and email parser for
> your SGE, Unless you can't use existing implementations you need to come up
> with your own implementations of these interfaces.
>
> 2. Don't modify thrift generated file, instead add your new
> ResourceJobManagerType to
> airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift
> file and generate thrift classes.
>
> 3. Come up with SGE JobConfiguration class, and modify
> org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to
> provide it. You should have xslt template for your SGE script file.
>
> let us know if you hit any issue while you following these steps.
>
> Thanks,
> Shameera.
>
> On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <kamalasini@gmail.com
> <ma...@gmail.com>> wrote:
> Hi Eric,
>
> To understand what you did, you added a new ResourceJobManagerType in the
> thrift definition as "SGE" ? I assume you generate thrift stubs after
> adding it. For that, you need to have thrift 0.93 installed in your system.
>
> Thanks..
> Chathuri
>
> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <eric.kuijt@wur.nl<mailto:
> eric.kuijt@wur.nl>> wrote:
> Hey all, I am playing around with airavata and when I can run jobs on my
> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
> the Executing state.
>
> Now I try to make my own SGEJobConfiguration class.
> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>
> - add SGE in
> /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>
> - add SGE in
> /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>
> - add SGE to
> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>
> What should the next step be? if I Wanted to create my own SGEOutputparser?
> PGA now gives me a Required field 'resourceJobManagerType' is unset! when
> I select SGE in the list.
>
> Thanks,
>
> Eric
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
> Web: http://www.cvi.wur.nl<http://www.cvi.wur.nl/>
>
> --
> Shameera Rathnayaka
>
>
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl
Web: http://www.cvi.wur.nl
Re: Adding a SGE resourceJobManagerType
Posted by Suresh Marru <sm...@apache.org>.
SGE - Sun Grid Engine is the old variant of Univa Grid Engine, so you should pick UGE in Airavata and should be good to go. SGE/UGE implement PBS protocol so parsing should be similar. Lonestar4 used to have UGE, so email parsing should be similar to that.
Suresh
> On Feb 24, 2016, at 10:17 AM, Shameera Rathnayaka <sh...@gmail.com> wrote:
>
> Hi Eric,
>
> There are couple of things you need to do to add a new jobmanager type.
>
> 1. Configure your SGE resource type in gfac yaml config file (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config). As you can see you need to provide an output parser and email parser for your SGE, Unless you can't use existing implementations you need to come up with your own implementations of these interfaces.
>
> 2. Don't modify thrift generated file, instead add your new ResourceJobManagerType to airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift file and generate thrift classes.
>
> 3. Come up with SGE JobConfiguration class, and modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to provide it. You should have xslt template for your SGE script file.
>
> let us know if you hit any issue while you following these steps.
>
> Thanks,
> Shameera.
>
> On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <kamalasini@gmail.com <ma...@gmail.com>> wrote:
> Hi Eric,
>
> To understand what you did, you added a new ResourceJobManagerType in the thrift definition as "SGE" ? I assume you generate thrift stubs after adding it. For that, you need to have thrift 0.93 installed in your system.
>
> Thanks..
> Chathuri
>
> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <eric.kuijt@wur.nl <ma...@wur.nl>> wrote:
> Hey all, I am playing around with airavata and when I can run jobs on my SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave the Executing state.
>
> Now I try to make my own SGEJobConfiguration class.
> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>
> - add SGE in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>
> - add SGE in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>
> - add SGE to airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>
> What should the next step be? if I Wanted to create my own SGEOutputparser?
> PGA now gives me a Required field 'resourceJobManagerType' is unset! when I select SGE in the list.
>
> Thanks,
>
> Eric
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391 <>
> Fax: +31-(0)320-238153 <>
> E-mail: eric.kuijt@wur.nl <ma...@wur.nl>
> Web: http://www.cvi.wur.nl <http://www.cvi.wur.nl/>
> --
> Shameera Rathnayaka
Re: Adding a SGE resourceJobManagerType
Posted by Shameera Rathnayaka <sh...@gmail.com>.
Hi Eric,
There are couple of things you need to do to add a new jobmanager type.
1. Configure your SGE resource type in gfac yaml config file
(airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config).
As you can see you need to provide an output parser and email parser for
your SGE, Unless you can't use existing implementations you need to come up
with your own implementations of these interfaces.
2. Don't modify thrift generated file, instead add your new
ResourceJobManagerType to
airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift
file and generate thrift classes.
3. Come up with SGE JobConfiguration class, and
modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration
method to provide it. You should have xslt template for your SGE script
file.
let us know if you hit any issue while you following these steps.
Thanks,
Shameera.
On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <ka...@gmail.com>
wrote:
> Hi Eric,
>
> To understand what you did, you added a new ResourceJobManagerType in the
> thrift definition as "SGE" ? I assume you generate thrift stubs after
> adding it. For that, you need to have thrift 0.93 installed in your
> system.
>
> Thanks..
> Chathuri
>
> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl> wrote:
>
>> Hey all, I am playing around with airavata and when I can run jobs on my
>> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
>> the Executing state.
>>
>> Now I try to make my own SGEJobConfiguration class.
>> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>>
>> - add SGE
>> in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>>
>> - add SGE
>> in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>>
>> - add SGE to
>> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>>
>> What should the next step be? if I Wanted to create my own
>> SGEOutputparser?
>> PGA now gives me a Required field 'resourceJobManagerType' is unset! when
>> I select SGE in the list.
>>
>> Thanks,
>>
>> Eric
>>
>>
>> --
>> Central Veterinary Institute of Wageningen UR (CVI)
>> Department of Infection Biology
>> PO box 65, 8200 AB Lelystad, NL
>> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>>
>> Tel: +31-(0)320-293391
>> Fax: +31-(0)320-238153
>> E-mail: eric.kuijt@wur.nl
>> Web: http://www.cvi.wur.nl
>>
>
> --
Shameera Rathnayaka
Re: Adding a SGE resourceJobManagerType
Posted by Chathuri Wimalasena <ka...@gmail.com>.
Hi Eric,
You need to add your XSLT file in
airavata/distribution/src/main/assembly/bin-assembly.xml. Otherwise it will
not be added to bin directory.
<fileSet>
<directory>${project.build.directory}/conf</directory>
<outputDirectory>bin</outputDirectory>
<includes>
<include>airavata-server.properties</include>
<include>zoo.cfg</include>
<include>registry.properties</include>
<include>log4j.properties</include>
<include>host.xml</include>
<include>persistence.xml</include>
<include>provenance.sql</include>
<include>gfac-config.yaml</include>
<include>PBSTemplate.xslt</include>
<include>SLURMTemplate.xslt</include>
<include>LSFTemplate.xslt</include>
<include>UGETemplate.xslt</include>
<include>ForkTemplate.xslt</include>
<include>gsissh.properties</include>
<include>airavata.jks</include>
<include>client_truststore.jks</include>
<include>airavata-default-xacml-policy.xml</include>
</includes>
</fileSet>
Thanks..
Chathuri
On Fri, Feb 26, 2016 at 10:56 AM, Pierce, Marlon <ma...@iu.edu> wrote:
> Hi Eric, is the path "/home/path/to/“ in your message below a verbatim
> error message, or did you change to remove your specific path?
>
> From: Eric Kuyt <er...@wur.nl>
> Reply-To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Date: Friday, February 26, 2016 at 10:52 AM
> To: "dev@airavata.apache.org" <de...@airavata.apache.org>
> Subject: Re: Adding a SGE resourceJobManagerType
>
> Hey All,
>
> I'm almost there! Ran the three steps (Only difference with UGE is -pe in
> the xslt file to build the pbs) so I use the UGE parse/config classes. Love
> thrift btw!
>
> But now, when I run My newly generated SGE it seems not to find the xslt
> file...
>
> error message is.
>
> /home/path/to/apache-airavata-server-0.16-SNAPSHOT/bin/file:/home/path/to/
> !/SGETemplate.xslt (No such file or directory)
>
> airavata-gfac-impl-0.16-SNAPSHOT.jar, seems to contain the xslt.
>
> jar -tf lib/airavata-gfac-impl-0.16-SNAPSHOT.jar |grep Template
> SGETemplate.xslt
> PBSTemplate.xslt
> UGETemplate.xslt
> SLURMTemplate.xslt
> ForkTemplate.xslt
> LSFTemplate.xslt
>
>
> The path seems to be printed double, Could this be the problem? PBS
> doesn't seem to have this problem.
>
> Thanks!
>
> Eric
>
> On 24 February 2016 at 16:17, Shameera Rathnayaka <sh...@gmail.com>
> wrote:
>
>> Hi Eric,
>>
>> There are couple of things you need to do to add a new jobmanager type.
>>
>> 1. Configure your SGE resource type in gfac yaml config file
>> (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config).
>> As you can see you need to provide an output parser and email parser for
>> your SGE, Unless you can't use existing implementations you need to come up
>> with your own implementations of these interfaces.
>>
>> 2. Don't modify thrift generated file, instead add your new
>> ResourceJobManagerType to
>> airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift
>> file and generate thrift classes.
>>
>> 3. Come up with SGE JobConfiguration class, and modify
>> org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to
>> provide it. You should have xslt template for your SGE script file.
>>
>> let us know if you hit any issue while you following these steps.
>>
>> Thanks,
>> Shameera.
>>
>> On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <kamalasini@gmail.com
>> <ma...@gmail.com>> wrote:
>> Hi Eric,
>>
>> To understand what you did, you added a new ResourceJobManagerType in the
>> thrift definition as "SGE" ? I assume you generate thrift stubs after
>> adding it. For that, you need to have thrift 0.93 installed in your system.
>>
>> Thanks..
>> Chathuri
>>
>> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <eric.kuijt@wur.nl<mailto:
>> eric.kuijt@wur.nl>> wrote:
>> Hey all, I am playing around with airavata and when I can run jobs on my
>> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
>> the Executing state.
>>
>> Now I try to make my own SGEJobConfiguration class.
>> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>>
>> - add SGE in
>> /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>>
>> - add SGE in
>> /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>>
>> - add SGE to
>> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>>
>> What should the next step be? if I Wanted to create my own
>> SGEOutputparser?
>> PGA now gives me a Required field 'resourceJobManagerType' is unset! when
>> I select SGE in the list.
>>
>> Thanks,
>>
>> Eric
>>
>>
>> --
>> Central Veterinary Institute of Wageningen UR (CVI)
>> Department of Infection Biology
>> PO box 65, 8200 AB Lelystad, NL
>> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>>
>> Tel: +31-(0)320-293391
>> Fax: +31-(0)320-238153
>> E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
>> Web: http://www.cvi.wur.nl
>>
>> --
>> Shameera Rathnayaka
>>
>
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.kuijt@wur.nl
> Web: http://www.cvi.wur.nl
>
Re: Adding a SGE resourceJobManagerType
Posted by "Pierce, Marlon" <ma...@iu.edu>.
Hi Eric, is the path "/home/path/to/“ in your message below a verbatim error message, or did you change to remove your specific path?
From: Eric Kuyt <er...@wur.nl>>
Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Date: Friday, February 26, 2016 at 10:52 AM
To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
Subject: Re: Adding a SGE resourceJobManagerType
Hey All,
I'm almost there! Ran the three steps (Only difference with UGE is -pe in the xslt file to build the pbs) so I use the UGE parse/config classes. Love thrift btw!
But now, when I run My newly generated SGE it seems not to find the xslt file...
error message is.
/home/path/to/apache-airavata-server-0.16-SNAPSHOT/bin/file:/home/path/to/
!/SGETemplate.xslt (No such file or directory)
airavata-gfac-impl-0.16-SNAPSHOT.jar, seems to contain the xslt.
jar -tf lib/airavata-gfac-impl-0.16-SNAPSHOT.jar |grep Template
SGETemplate.xslt
PBSTemplate.xslt
UGETemplate.xslt
SLURMTemplate.xslt
ForkTemplate.xslt
LSFTemplate.xslt
The path seems to be printed double, Could this be the problem? PBS doesn't seem to have this problem.
Thanks!
Eric
On 24 February 2016 at 16:17, Shameera Rathnayaka <sh...@gmail.com>> wrote:
Hi Eric,
There are couple of things you need to do to add a new jobmanager type.
1. Configure your SGE resource type in gfac yaml config file (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config). As you can see you need to provide an output parser and email parser for your SGE, Unless you can't use existing implementations you need to come up with your own implementations of these interfaces.
2. Don't modify thrift generated file, instead add your new ResourceJobManagerType to airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift file and generate thrift classes.
3. Come up with SGE JobConfiguration class, and modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to provide it. You should have xslt template for your SGE script file.
let us know if you hit any issue while you following these steps.
Thanks,
Shameera.
On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <ka...@gmail.com>>> wrote:
Hi Eric,
To understand what you did, you added a new ResourceJobManagerType in the thrift definition as "SGE" ? I assume you generate thrift stubs after adding it. For that, you need to have thrift 0.93 installed in your system.
Thanks..
Chathuri
On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl>>> wrote:
Hey all, I am playing around with airavata and when I can run jobs on my SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave the Executing state.
Now I try to make my own SGEJobConfiguration class.
Created the class plus outputParser in org.apache.airavata.gfac.impl.job
- add SGE in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
- add SGE in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
- add SGE to airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
What should the next step be? if I Wanted to create my own SGEOutputparser?
PGA now gives me a Required field 'resourceJobManagerType' is unset! when I select SGE in the list.
Thanks,
Eric
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391<tel:%2B31-%280%29320-293391>
Fax: +31-(0)320-238153<tel:%2B31-%280%29320-238153>
E-mail: eric.kuijt@wur.nl<ma...@wur.nl>>
Web: http://www.cvi.wur.nl
--
Shameera Rathnayaka
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
Web: http://www.cvi.wur.nl
Re: Adding a SGE resourceJobManagerType
Posted by Chathuri Wimalasena <ka...@gmail.com>.
Can you see your xslt in <airavata-16-SNAPSHOT>/bin directory ?
On Fri, Feb 26, 2016 at 11:03 AM, Eric Kuijt <er...@gmail.com> wrote:
> I did depersonalize the path for a bit..
> But it is double in the logs.
>
> On Fri Feb 26 16:56:31 2016 GMT+0100, Pierce, Marlon wrote:
> > Hi Eric, is the path "/home/path/to/“ in your message below a verbatim
> error message, or did you change to remove your specific path?
> >
> > From: Eric Kuyt <er...@wur.nl>>
> > Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <
> dev@airavata.apache.org<ma...@airavata.apache.org>>
> > Date: Friday, February 26, 2016 at 10:52 AM
> > To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <
> dev@airavata.apache.org<ma...@airavata.apache.org>>
> > Subject: Re: Adding a SGE resourceJobManagerType
> >
> > Hey All,
> >
> > I'm almost there! Ran the three steps (Only difference with UGE is -pe
> in the xslt file to build the pbs) so I use the UGE parse/config classes.
> Love thrift btw!
> >
> > But now, when I run My newly generated SGE it seems not to find the xslt
> file...
> >
> > error message is.
> >
> >
> /home/path/to/apache-airavata-server-0.16-SNAPSHOT/bin/file:/home/path/to/
> > !/SGETemplate.xslt (No such file or directory)
> >
> > airavata-gfac-impl-0.16-SNAPSHOT.jar, seems to contain the xslt.
> >
> > jar -tf lib/airavata-gfac-impl-0.16-SNAPSHOT.jar |grep Template
> > SGETemplate.xslt
> > PBSTemplate.xslt
> > UGETemplate.xslt
> > SLURMTemplate.xslt
> > ForkTemplate.xslt
> > LSFTemplate.xslt
> >
> >
> > The path seems to be printed double, Could this be the problem? PBS
> doesn't seem to have this problem.
> >
> > Thanks!
> >
> > Eric
> >
> > On 24 February 2016 at 16:17, Shameera Rathnayaka <
> shameerainfo@gmail.com<ma...@gmail.com>> wrote:
> > Hi Eric,
> >
> > There are couple of things you need to do to add a new jobmanager type.
> >
> > 1. Configure your SGE resource type in gfac yaml config file
> (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config).
> As you can see you need to provide an output parser and email parser for
> your SGE, Unless you can't use existing implementations you need to come up
> with your own implementations of these interfaces.
> >
> > 2. Don't modify thrift generated file, instead add your new
> ResourceJobManagerType to
> airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift
> file and generate thrift classes.
> >
> > 3. Come up with SGE JobConfiguration class, and modify
> org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to
> provide it. You should have xslt template for your SGE script file.
> >
> > let us know if you hit any issue while you following these steps.
> >
> > Thanks,
> > Shameera.
> >
> > On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <
> kamalasini@gmail.com<ma...@gmail.com><mailto:
> kamalasini@gmail.com<ma...@gmail.com>>> wrote:
> > Hi Eric,
> >
> > To understand what you did, you added a new ResourceJobManagerType in
> the thrift definition as "SGE" ? I assume you generate thrift stubs after
> adding it. For that, you need to have thrift 0.93 installed in your system.
> >
> > Thanks..
> > Chathuri
> >
> > On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <eric.kuijt@wur.nl<mailto:
> eric.kuijt@wur.nl><ma...@wur.nl>>>
> wrote:
> > Hey all, I am playing around with airavata and when I can run jobs on my
> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
> the Executing state.
> >
> > Now I try to make my own SGEJobConfiguration class.
> > Created the class plus outputParser in org.apache.airavata.gfac.impl.job
> >
> > - add SGE in
> /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
> >
> > - add SGE in
> /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
> >
> > - add SGE to
> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
> >
> > What should the next step be? if I Wanted to create my own
> SGEOutputparser?
> > PGA now gives me a Required field 'resourceJobManagerType' is unset!
> when I select SGE in the list.
> >
> > Thanks,
> >
> > Eric
> >
> >
> > --
> > Central Veterinary Institute of Wageningen UR (CVI)
> > Department of Infection Biology
> > PO box 65, 8200 AB Lelystad, NL
> > Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
> >
> > Tel: +31-(0)320-293391<tel:%2B31-%280%29320-293391>
> > Fax: +31-(0)320-238153<tel:%2B31-%280%29320-238153>
> > E-mail: eric.kuijt@wur.nl<ma...@wur.nl><mailto:
> eric.kuijt@wur.nl<ma...@wur.nl>>
> > Web: http://www.cvi.wur.nl
> >
> > --
> > Shameera Rathnayaka
> >
> >
> >
> > --
> > Central Veterinary Institute of Wageningen UR (CVI)
> > Department of Infection Biology
> > PO box 65, 8200 AB Lelystad, NL
> > Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
> >
> > Tel: +31-(0)320-293391
> > Fax: +31-(0)320-238153
> > E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
> > Web: http://www.cvi.wur.nl
> >
>
> --
> Sent from my Jolla
Re: Adding a SGE resourceJobManagerType
Posted by Eric Kuijt <er...@gmail.com>.
I did depersonalize the path for a bit..
But it is double in the logs.
On Fri Feb 26 16:56:31 2016 GMT+0100, Pierce, Marlon wrote:
> Hi Eric, is the path "/home/path/to/“ in your message below a verbatim error message, or did you change to remove your specific path?
>
> From: Eric Kuyt <er...@wur.nl>>
> Reply-To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
> Date: Friday, February 26, 2016 at 10:52 AM
> To: "dev@airavata.apache.org<ma...@airavata.apache.org>" <de...@airavata.apache.org>>
> Subject: Re: Adding a SGE resourceJobManagerType
>
> Hey All,
>
> I'm almost there! Ran the three steps (Only difference with UGE is -pe in the xslt file to build the pbs) so I use the UGE parse/config classes. Love thrift btw!
>
> But now, when I run My newly generated SGE it seems not to find the xslt file...
>
> error message is.
>
> /home/path/to/apache-airavata-server-0.16-SNAPSHOT/bin/file:/home/path/to/
> !/SGETemplate.xslt (No such file or directory)
>
> airavata-gfac-impl-0.16-SNAPSHOT.jar, seems to contain the xslt.
>
> jar -tf lib/airavata-gfac-impl-0.16-SNAPSHOT.jar |grep Template
> SGETemplate.xslt
> PBSTemplate.xslt
> UGETemplate.xslt
> SLURMTemplate.xslt
> ForkTemplate.xslt
> LSFTemplate.xslt
>
>
> The path seems to be printed double, Could this be the problem? PBS doesn't seem to have this problem.
>
> Thanks!
>
> Eric
>
> On 24 February 2016 at 16:17, Shameera Rathnayaka <sh...@gmail.com>> wrote:
> Hi Eric,
>
> There are couple of things you need to do to add a new jobmanager type.
>
> 1. Configure your SGE resource type in gfac yaml config file (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config). As you can see you need to provide an output parser and email parser for your SGE, Unless you can't use existing implementations you need to come up with your own implementations of these interfaces.
>
> 2. Don't modify thrift generated file, instead add your new ResourceJobManagerType to airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift file and generate thrift classes.
>
> 3. Come up with SGE JobConfiguration class, and modify org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to provide it. You should have xslt template for your SGE script file.
>
> let us know if you hit any issue while you following these steps.
>
> Thanks,
> Shameera.
>
> On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <ka...@gmail.com>>> wrote:
> Hi Eric,
>
> To understand what you did, you added a new ResourceJobManagerType in the thrift definition as "SGE" ? I assume you generate thrift stubs after adding it. For that, you need to have thrift 0.93 installed in your system.
>
> Thanks..
> Chathuri
>
> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl>>> wrote:
> Hey all, I am playing around with airavata and when I can run jobs on my SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave the Executing state.
>
> Now I try to make my own SGEJobConfiguration class.
> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>
> - add SGE in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>
> - add SGE in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>
> - add SGE to airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>
> What should the next step be? if I Wanted to create my own SGEOutputparser?
> PGA now gives me a Required field 'resourceJobManagerType' is unset! when I select SGE in the list.
>
> Thanks,
>
> Eric
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391<tel:%2B31-%280%29320-293391>
> Fax: +31-(0)320-238153<tel:%2B31-%280%29320-238153>
> E-mail: eric.kuijt@wur.nl<ma...@wur.nl>>
> Web: http://www.cvi.wur.nl
>
> --
> Shameera Rathnayaka
>
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
> Web: http://www.cvi.wur.nl
>
--
Sent from my Jolla
Re: Adding a SGE resourceJobManagerType
Posted by Eric Kuyt <er...@wur.nl>.
Hey All,
I'm almost there! Ran the three steps (Only difference with UGE is -pe in
the xslt file to build the pbs) so I use the UGE parse/config classes. Love
thrift btw!
But now, when I run My newly generated SGE it seems not to find the xslt
file...
error message is.
/home/path/to/apache-airavata-server-0.16-SNAPSHOT/bin/file:/home/path/to/
!/SGETemplate.xslt (No such file or directory)
airavata-gfac-impl-0.16-SNAPSHOT.jar, seems to contain the xslt.
jar -tf lib/airavata-gfac-impl-0.16-SNAPSHOT.jar |grep Template
SGETemplate.xslt
PBSTemplate.xslt
UGETemplate.xslt
SLURMTemplate.xslt
ForkTemplate.xslt
LSFTemplate.xslt
The path seems to be printed double, Could this be the problem? PBS doesn't
seem to have this problem.
Thanks!
Eric
On 24 February 2016 at 16:17, Shameera Rathnayaka <sh...@gmail.com>
wrote:
> Hi Eric,
>
> There are couple of things you need to do to add a new jobmanager type.
>
> 1. Configure your SGE resource type in gfac yaml config file
> (airavata/modules/configuration/server/src/main/resources/gfac-config.yaml.config).
> As you can see you need to provide an output parser and email parser for
> your SGE, Unless you can't use existing implementations you need to come up
> with your own implementations of these interfaces.
>
> 2. Don't modify thrift generated file, instead add your new
> ResourceJobManagerType to
> airavata/thrift-interface-descriptions/data-models/resource-catalog-models/compute_resource_model.thrift
> file and generate thrift classes.
>
> 3. Come up with SGE JobConfiguration class, and modify
> org.apache.airavata.gfac.impl.Factory#getJobManagerConfiguration method to
> provide it. You should have xslt template for your SGE script file.
>
> let us know if you hit any issue while you following these steps.
>
> Thanks,
> Shameera.
>
> On Wed, Feb 24, 2016 at 9:53 AM Chathuri Wimalasena <kamalasini@gmail.com
> <ma...@gmail.com>> wrote:
> Hi Eric,
>
> To understand what you did, you added a new ResourceJobManagerType in the
> thrift definition as "SGE" ? I assume you generate thrift stubs after
> adding it. For that, you need to have thrift 0.93 installed in your system.
>
> Thanks..
> Chathuri
>
> On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <eric.kuijt@wur.nl<mailto:
> eric.kuijt@wur.nl>> wrote:
> Hey all, I am playing around with airavata and when I can run jobs on my
> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
> the Executing state.
>
> Now I try to make my own SGEJobConfiguration class.
> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>
> - add SGE in
> /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>
> - add SGE in
> /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>
> - add SGE to
> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>
> What should the next step be? if I Wanted to create my own SGEOutputparser?
> PGA now gives me a Required field 'resourceJobManagerType' is unset! when
> I select SGE in the list.
>
> Thanks,
>
> Eric
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.kuijt@wur.nl<ma...@wur.nl>
> Web: http://www.cvi.wur.nl
>
> --
> Shameera Rathnayaka
>
--
Central Veterinary Institute of Wageningen UR (CVI)
Department of Infection Biology
PO box 65, 8200 AB Lelystad, NL
Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
Tel: +31-(0)320-293391
Fax: +31-(0)320-238153
E-mail: eric.kuijt@wur.nl
Web: http://www.cvi.wur.nl
Re: Adding a SGE resourceJobManagerType
Posted by Chathuri Wimalasena <ka...@gmail.com>.
Hi Eric,
To understand what you did, you added a new ResourceJobManagerType in the
thrift definition as "SGE" ? I assume you generate thrift stubs after
adding it. For that, you need to have thrift 0.93 installed in your
system.
Thanks..
Chathuri
On Wed, Feb 24, 2016 at 9:03 AM, Eric Kuyt <er...@wur.nl> wrote:
> Hey all, I am playing around with airavata and when I can run jobs on my
> SGE using PBS but the parsing of qsub / qstat fails. So my jobs never leave
> the Executing state.
>
> Now I try to make my own SGEJobConfiguration class.
> Created the class plus outputParser in org.apache.airavata.gfac.impl.job
>
> - add SGE
> in /airavata-gfac-impl/src/main/java/org/apache/airavata/gfac/impl/Factory.java
>
> - add SGE
> in /airavata-data-models/src/main/java/org/apache/airavata/model/appcatalog/computeresource/ResourceJobManagerType.java
>
> - add SGE to
> airavata-php-gateway/app/libraries/Airavata/Model/Job/Types.php in PGA.
>
> What should the next step be? if I Wanted to create my own SGEOutputparser?
> PGA now gives me a Required field 'resourceJobManagerType' is unset! when
> I select SGE in the list.
>
> Thanks,
>
> Eric
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel: +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.kuijt@wur.nl
> Web: http://www.cvi.wur.nl
>