You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> on 2016/08/16 23:01:44 UTC
Resource Manager documentation
Hi all,
I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
I executed what I thought were the necessary steps but I get the following error:
FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
thanks, Luca
Re: Resource Manager documentation
Posted by "Ramirez, Paul M (398M)" <pa...@jpl.nasa.gov>.
Nice catch! That checked in example must be using the resource manager in local mode. I know we’ve done that in distributed mode… Can’t seem to find a public reference to that at the moment.
--Paul
======================================================================
Paul Ramirez - Group Supervisor
Computer Science for Data Intensive Applications (398M)
NASA - Jet Propulsion Laboratory
4800 Oak Grove Dr.
Pasadena, CA 91109 USA
Mailstop: 158-242
Office: 818-354-1015
Cell: 818-395-8194
======================================================================
On 8/17/16, 11:54 AM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>Hi Paul,
> thanks for the pointer, but it seems like the Resource Manager URL is disabled in this project:
>
># set this if you want the workflow manager to submit jobs through the resource mgr
>#org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9300
>
>Or maybe I don’t understand the configuration well enough. No problem, I’ll try to debug the problem.
>
>thanks, L
>
>> On Aug 17, 2016, at 8:40 AM, Ramirez, Paul M (398M) <pa...@jpl.nasa.gov> wrote:
>>
>> Luca,
>>
>> You may want to take a look at DRAT. It is a good example of a system built on top of OODT with working configs. Since it was built with Apache OODT RADIX initially the configs will be checked into the normal places.
>>
>> Top Level Repo:
>> https://github.com/chrismattmann/drat
>>
>> Workflow Config:
>> https://github.com/chrismattmann/drat/tree/master/workflow/src/main/resources
>>
>> Resource Manager Config:
>> https://github.com/chrismattmann/drat/tree/master/resmgr/src/main/resources
>>
>>
>> HTH,
>> Paul
>>
>>
>> ======================================================================
>> Paul Ramirez - Group Supervisor
>> Computer Science for Data Intensive Applications (398M)
>> NASA - Jet Propulsion Laboratory
>> 4800 Oak Grove Dr.
>> Pasadena, CA 91109 USA
>> Mailstop: 158-242
>> Office: 818-354-1015
>> Cell: 818-395-8194
>> ======================================================================
>>
>> On 8/17/16, 5:26 AM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>>
>>> Hi Chris,
>>> I sure did… these are the steps I took:
>>>
>>> o edit workflow.properties:
>>> org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9002/
>>>
>>> o start batch stub on port 2001:
>>>
>>> ./batch_stub 2001
>>>
>>> Are there any other changes I need to make to have RM work out of the box - for example, to resource.properties ?
>>>
>>> thanks a lot,
>>> Luca
>>>
>>>> On Aug 16, 2016, at 6:08 PM, Chris Mattmann <ma...@apache.org> wrote:
>>>>
>>>> Hi Luca,
>>>>
>>>> Did you start the batch stub on port 2001 (the default one?)
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>>
>>>>
>>>>
>>>> On 8/16/16, 4:01 PM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>>>>
>>>> Hi all,
>>>> I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
>>>>
>>>> I executed what I thought were the necessary steps but I get the following error:
>>>>
>>>> FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
>>>> Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
>>>> WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
>>>>
>>>> thanks, Luca
>>>>
>>>>
>>>>
>>>
>>
>
Re: Resource Manager documentation
Posted by "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov>.
Hi Paul,
thanks for the pointer, but it seems like the Resource Manager URL is disabled in this project:
# set this if you want the workflow manager to submit jobs through the resource mgr
#org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9300
Or maybe I don’t understand the configuration well enough. No problem, I’ll try to debug the problem.
thanks, L
> On Aug 17, 2016, at 8:40 AM, Ramirez, Paul M (398M) <pa...@jpl.nasa.gov> wrote:
>
> Luca,
>
> You may want to take a look at DRAT. It is a good example of a system built on top of OODT with working configs. Since it was built with Apache OODT RADIX initially the configs will be checked into the normal places.
>
> Top Level Repo:
> https://github.com/chrismattmann/drat
>
> Workflow Config:
> https://github.com/chrismattmann/drat/tree/master/workflow/src/main/resources
>
> Resource Manager Config:
> https://github.com/chrismattmann/drat/tree/master/resmgr/src/main/resources
>
>
> HTH,
> Paul
>
>
> ======================================================================
> Paul Ramirez - Group Supervisor
> Computer Science for Data Intensive Applications (398M)
> NASA - Jet Propulsion Laboratory
> 4800 Oak Grove Dr.
> Pasadena, CA 91109 USA
> Mailstop: 158-242
> Office: 818-354-1015
> Cell: 818-395-8194
> ======================================================================
>
> On 8/17/16, 5:26 AM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>
>> Hi Chris,
>> I sure did… these are the steps I took:
>>
>> o edit workflow.properties:
>> org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9002/
>>
>> o start batch stub on port 2001:
>>
>> ./batch_stub 2001
>>
>> Are there any other changes I need to make to have RM work out of the box - for example, to resource.properties ?
>>
>> thanks a lot,
>> Luca
>>
>>> On Aug 16, 2016, at 6:08 PM, Chris Mattmann <ma...@apache.org> wrote:
>>>
>>> Hi Luca,
>>>
>>> Did you start the batch stub on port 2001 (the default one?)
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>>
>>>
>>> On 8/16/16, 4:01 PM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>>>
>>> Hi all,
>>> I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
>>>
>>> I executed what I thought were the necessary steps but I get the following error:
>>>
>>> FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
>>> Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
>>> WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
>>>
>>> thanks, Luca
>>>
>>>
>>>
>>
>
Re: Resource Manager documentation
Posted by "Ramirez, Paul M (398M)" <pa...@jpl.nasa.gov>.
Luca,
You may want to take a look at DRAT. It is a good example of a system built on top of OODT with working configs. Since it was built with Apache OODT RADIX initially the configs will be checked into the normal places.
Top Level Repo:
https://github.com/chrismattmann/drat
Workflow Config:
https://github.com/chrismattmann/drat/tree/master/workflow/src/main/resources
Resource Manager Config:
https://github.com/chrismattmann/drat/tree/master/resmgr/src/main/resources
HTH,
Paul
======================================================================
Paul Ramirez - Group Supervisor
Computer Science for Data Intensive Applications (398M)
NASA - Jet Propulsion Laboratory
4800 Oak Grove Dr.
Pasadena, CA 91109 USA
Mailstop: 158-242
Office: 818-354-1015
Cell: 818-395-8194
======================================================================
On 8/17/16, 5:26 AM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>Hi Chris,
> I sure did… these are the steps I took:
>
>o edit workflow.properties:
>org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9002/
>
>o start batch stub on port 2001:
>
> ./batch_stub 2001
>
>Are there any other changes I need to make to have RM work out of the box - for example, to resource.properties ?
>
>thanks a lot,
>Luca
>
>> On Aug 16, 2016, at 6:08 PM, Chris Mattmann <ma...@apache.org> wrote:
>>
>> Hi Luca,
>>
>> Did you start the batch stub on port 2001 (the default one?)
>>
>> Cheers,
>> Chris
>>
>>
>>
>>
>> On 8/16/16, 4:01 PM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>>
>> Hi all,
>> I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
>>
>> I executed what I thought were the necessary steps but I get the following error:
>>
>> FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
>> Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
>> WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
>>
>> thanks, Luca
>>
>>
>>
>
Re: Resource Manager documentation
Posted by "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov>.
Hi Chris,
I sure did… these are the steps I took:
o edit workflow.properties:
org.apache.oodt.cas.workflow.engine.resourcemgr.url=http://localhost:9002/
o start batch stub on port 2001:
./batch_stub 2001
Are there any other changes I need to make to have RM work out of the box - for example, to resource.properties ?
thanks a lot,
Luca
> On Aug 16, 2016, at 6:08 PM, Chris Mattmann <ma...@apache.org> wrote:
>
> Hi Luca,
>
> Did you start the batch stub on port 2001 (the default one?)
>
> Cheers,
> Chris
>
>
>
>
> On 8/16/16, 4:01 PM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
>
> Hi all,
> I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
>
> I executed what I thought were the necessary steps but I get the following error:
>
> FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
> Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
> WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
>
> thanks, Luca
>
>
>
Re: Resource Manager documentation
Posted by Chris Mattmann <ma...@apache.org>.
Hi Luca,
Did you start the batch stub on port 2001 (the default one?)
Cheers,
Chris
On 8/16/16, 4:01 PM, "Cinquini, Luca (398G)" <Lu...@jpl.nasa.gov> wrote:
Hi all,
I should probably know this but I don’t… do we have a tutorial on how to connect the Resource manager to the Workflow Manager ?
I executed what I thought were the necessary steps but I get the following error:
FINEST: [{job.queueName=high, job.instanceClassName=org.apache.oodt.cas.workflow.structs.TaskJob, job.name=urn:edrn:LabcasTestTask, job.id=, job.status=, job.load=2, job.inputClassName=org.apache.oodt.cas.workflow.structs.TaskJobInput}, {task.instance.class=org.apache.oodt.cas.pge.StdPGETaskInstance, task.config={PGETask_ConfigFilePath=/usr/local/labcas_home/workflows/labcas-test/pge-configs/labcas-test-task-config.pgeconfig.xml, PCS_ClientTransferServiceFactory=org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory, PCS_ActionRepoFile=file:/usr/local/labcas_home/cas-crawler/policy/crawler-config.xml, PCS_MetFileExtension=met, PGETask_DumpMetadata=true, PCS_WorkflowManagerUrl=http://localhost:9001, PCS_FileManagerUrl=http://localhost:9000, PGETask_Name=LabcasTestTask}, task.metadata={WorkflowInstId=[a195a9e9-63fb-11e6-8305-6d5e443f934e], TaskId=[urn:edrn:LabcasTestTask], experiment=[1], species=[snakes], WorkflowName=[LabcasTestWorkflow], ProcessingNode=[LMC-038261.local], location=[LAB01], WorkflowManagerUrl=[http://LMC-038261.local:9001], WorkflowId=[urn:edrn:LabcasTestWorkflow], JobId=[a195a9e9-63fb-11e6-8305-6d5e443f934e]}}]
Aug 16, 2016 3:51:48 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread run
WARNING: Job execution exception using resource manager to execute job: Message: Failure writing request
thanks, Luca