You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov> on 2011/10/25 01:52:36 UTC
Sharing workflow task metadata when using a resource manager
Hi all,
I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
Thanks!
rishi
Re: Sharing workflow task metadata when using a resource manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
No problem!
Cheers,
Chris
On Oct 27, 2011, at 6:14 PM, Verma, Rishi (388J) wrote:
> Hey Chris,
>
> I tried 0.3, and it worked just fine (generated workflow met passed down to other tasks when using resmgr batchstubs). Thanks for the suggestion!
>
> rishi
>
> On Oct 26, 2011, at 6:44 PM, Mattmann, Chris A (388J) wrote:
>
>> Hi Rishi,
>>
>> Thanks. Yep that's the problem below (and why Jenkins has been screaming at me for weeks). I broke the build
>> a few weeks ago when I committed portions of OODT-310. I want to wrap up OODT-310 and need to get to it,
>> but in the meanwhile, if you can use a revision prior to those commits (or better yet, a revision right around
>> when I wrapped up OODT-70), or even 0.3, that will get you on the right track for now.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On Oct 26, 2011, at 3:17 PM, Verma, Rishi (388J) wrote:
>>
>>> Hi Chris,
>>>
>>> Thanks for the code snippet.
>>>
>>> I'm running my XmlRpcBatchStub on the same host as my workflow manager, so I don't think that is the issue.
>>>
>>> I looked at my XmlRpcBatchStub log however, and found the following exception being issued whenever one of my tasks was sent to this batch stub:
>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.ClassCastException: org.apache.oodt.cas.workflow.engine.SequentialProcessor cannot be cast to org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine$ThreadedProcessor
>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.updateMetadataForWorkflow(XmlRpcWorkflowManagerClient.java:424)
>>> at org.apache.oodt.cas.workflow.structs.TaskJob.updateMetadata(TaskJob.java:137)
>>> at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:113)
>>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>>> at java.lang.Thread.run(Thread.java:619)
>>>
>>> I'm using workflow-manager 0.4-SNAPSHOT (not the latest trunk). Off the bat, does anything look strange to you? It seems from the exception, that the workflow met is definitely not being properly passed to the batch stub.
>>>
>>> I know a lot of updates have been made to WM lately, so I tried to build the latest trunk version of 0.4-SNAPSHOT; however, it currently fails the 'TestXmlRpcWorkflowManagerClient' test with:
>>> java.text.ParseException: An exception occurred because the input date/time string was not at least 24 characters in length.
>>> at org.apache.oodt.commons.util.DateConvert.isoParse(DateConvert.java:165)
>>> at org.apache.oodt.cas.workflow.structs.WorkflowInstance.setCurrentTaskStartDateTimeIsoStr(WorkflowInstance.java:339)
>>>
>>> I'm going to look more into it and investigate, but maybe I will try deploying with 0.3 OODT and see if the problem persists.
>>>
>>> Thanks!
>>> rishi
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Oct 24, 2011, at 9:07 PM, Mattmann, Chris A (388J) wrote:
>>>
>>>> Hi Rishi,
>>>>
>>>> The metadata context should be persisted, based on updates made during the tasks themselves. If you have a look at
>>>> TaskJob [1], you'll see the last step in the execute(JobInput in) method is:
>>>>
>>>> {code}
>>>> // now we have to update the workflow manager with the metadata
>>>> // that may have been updated
>>>> updateMetadata(taskInput.getDynMetadata());
>>>> {code}
>>>>
>>>> which should call back to the parent workflow manager and update the workflow instance
>>>> with the current metadata. If you're not seeing this it's possible that your XmlRpcBatchStub
>>>> on your executing node cannot see back to the parent WM running on host:port.
>>>>
>>>> Is this the case?
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On Oct 24, 2011, at 4:52 PM, Verma, Rishi (388J) wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
>>>>>
>>>>> I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
>>>>>
>>>>> By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
>>>>>
>>>>> I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
>>>>>
>>>>> Thanks!
>>>>> rishi
>>>>>
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Sharing workflow task metadata when using a resource manager
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hey Chris,
I tried 0.3, and it worked just fine (generated workflow met passed down to other tasks when using resmgr batchstubs). Thanks for the suggestion!
rishi
On Oct 26, 2011, at 6:44 PM, Mattmann, Chris A (388J) wrote:
> Hi Rishi,
>
> Thanks. Yep that's the problem below (and why Jenkins has been screaming at me for weeks). I broke the build
> a few weeks ago when I committed portions of OODT-310. I want to wrap up OODT-310 and need to get to it,
> but in the meanwhile, if you can use a revision prior to those commits (or better yet, a revision right around
> when I wrapped up OODT-70), or even 0.3, that will get you on the right track for now.
>
> Thanks!
>
> Cheers,
> Chris
>
>
> On Oct 26, 2011, at 3:17 PM, Verma, Rishi (388J) wrote:
>
>> Hi Chris,
>>
>> Thanks for the code snippet.
>>
>> I'm running my XmlRpcBatchStub on the same host as my workflow manager, so I don't think that is the issue.
>>
>> I looked at my XmlRpcBatchStub log however, and found the following exception being issued whenever one of my tasks was sent to this batch stub:
>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.ClassCastException: org.apache.oodt.cas.workflow.engine.SequentialProcessor cannot be cast to org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine$ThreadedProcessor
>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.updateMetadataForWorkflow(XmlRpcWorkflowManagerClient.java:424)
>> at org.apache.oodt.cas.workflow.structs.TaskJob.updateMetadata(TaskJob.java:137)
>> at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:113)
>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>> at java.lang.Thread.run(Thread.java:619)
>>
>> I'm using workflow-manager 0.4-SNAPSHOT (not the latest trunk). Off the bat, does anything look strange to you? It seems from the exception, that the workflow met is definitely not being properly passed to the batch stub.
>>
>> I know a lot of updates have been made to WM lately, so I tried to build the latest trunk version of 0.4-SNAPSHOT; however, it currently fails the 'TestXmlRpcWorkflowManagerClient' test with:
>> java.text.ParseException: An exception occurred because the input date/time string was not at least 24 characters in length.
>> at org.apache.oodt.commons.util.DateConvert.isoParse(DateConvert.java:165)
>> at org.apache.oodt.cas.workflow.structs.WorkflowInstance.setCurrentTaskStartDateTimeIsoStr(WorkflowInstance.java:339)
>>
>> I'm going to look more into it and investigate, but maybe I will try deploying with 0.3 OODT and see if the problem persists.
>>
>> Thanks!
>> rishi
>>
>>
>>
>>
>>
>>
>> On Oct 24, 2011, at 9:07 PM, Mattmann, Chris A (388J) wrote:
>>
>>> Hi Rishi,
>>>
>>> The metadata context should be persisted, based on updates made during the tasks themselves. If you have a look at
>>> TaskJob [1], you'll see the last step in the execute(JobInput in) method is:
>>>
>>> {code}
>>> // now we have to update the workflow manager with the metadata
>>> // that may have been updated
>>> updateMetadata(taskInput.getDynMetadata());
>>> {code}
>>>
>>> which should call back to the parent workflow manager and update the workflow instance
>>> with the current metadata. If you're not seeing this it's possible that your XmlRpcBatchStub
>>> on your executing node cannot see back to the parent WM running on host:port.
>>>
>>> Is this the case?
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Oct 24, 2011, at 4:52 PM, Verma, Rishi (388J) wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
>>>>
>>>> I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
>>>>
>>>> By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
>>>>
>>>> I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
>>>>
>>>> Thanks!
>>>> rishi
>>>>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Sharing workflow task metadata when using a resource manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Rishi,
Thanks. Yep that's the problem below (and why Jenkins has been screaming at me for weeks). I broke the build
a few weeks ago when I committed portions of OODT-310. I want to wrap up OODT-310 and need to get to it,
but in the meanwhile, if you can use a revision prior to those commits (or better yet, a revision right around
when I wrapped up OODT-70), or even 0.3, that will get you on the right track for now.
Thanks!
Cheers,
Chris
On Oct 26, 2011, at 3:17 PM, Verma, Rishi (388J) wrote:
> Hi Chris,
>
> Thanks for the code snippet.
>
> I'm running my XmlRpcBatchStub on the same host as my workflow manager, so I don't think that is the issue.
>
> I looked at my XmlRpcBatchStub log however, and found the following exception being issued whenever one of my tasks was sent to this batch stub:
> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.ClassCastException: org.apache.oodt.cas.workflow.engine.SequentialProcessor cannot be cast to org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine$ThreadedProcessor
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.updateMetadataForWorkflow(XmlRpcWorkflowManagerClient.java:424)
> at org.apache.oodt.cas.workflow.structs.TaskJob.updateMetadata(TaskJob.java:137)
> at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:113)
> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
> at java.lang.Thread.run(Thread.java:619)
>
> I'm using workflow-manager 0.4-SNAPSHOT (not the latest trunk). Off the bat, does anything look strange to you? It seems from the exception, that the workflow met is definitely not being properly passed to the batch stub.
>
> I know a lot of updates have been made to WM lately, so I tried to build the latest trunk version of 0.4-SNAPSHOT; however, it currently fails the 'TestXmlRpcWorkflowManagerClient' test with:
> java.text.ParseException: An exception occurred because the input date/time string was not at least 24 characters in length.
> at org.apache.oodt.commons.util.DateConvert.isoParse(DateConvert.java:165)
> at org.apache.oodt.cas.workflow.structs.WorkflowInstance.setCurrentTaskStartDateTimeIsoStr(WorkflowInstance.java:339)
>
> I'm going to look more into it and investigate, but maybe I will try deploying with 0.3 OODT and see if the problem persists.
>
> Thanks!
> rishi
>
>
>
>
>
>
> On Oct 24, 2011, at 9:07 PM, Mattmann, Chris A (388J) wrote:
>
>> Hi Rishi,
>>
>> The metadata context should be persisted, based on updates made during the tasks themselves. If you have a look at
>> TaskJob [1], you'll see the last step in the execute(JobInput in) method is:
>>
>> {code}
>> // now we have to update the workflow manager with the metadata
>> // that may have been updated
>> updateMetadata(taskInput.getDynMetadata());
>> {code}
>>
>> which should call back to the parent workflow manager and update the workflow instance
>> with the current metadata. If you're not seeing this it's possible that your XmlRpcBatchStub
>> on your executing node cannot see back to the parent WM running on host:port.
>>
>> Is this the case?
>>
>> Cheers,
>> Chris
>>
>> On Oct 24, 2011, at 4:52 PM, Verma, Rishi (388J) wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
>>>
>>> I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
>>>
>>> By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
>>>
>>> I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
>>>
>>> Thanks!
>>> rishi
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Sharing workflow task metadata when using a resource manager
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hi Chris,
Thanks for the code snippet.
I'm running my XmlRpcBatchStub on the same host as my workflow manager, so I don't think that is the issue.
I looked at my XmlRpcBatchStub log however, and found the following exception being issued whenever one of my tasks was sent to this batch stub:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: java.lang.ClassCastException: org.apache.oodt.cas.workflow.engine.SequentialProcessor cannot be cast to org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngine$ThreadedProcessor
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManagerClient.updateMetadataForWorkflow(XmlRpcWorkflowManagerClient.java:424)
at org.apache.oodt.cas.workflow.structs.TaskJob.updateMetadata(TaskJob.java:137)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:113)
at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:619)
I'm using workflow-manager 0.4-SNAPSHOT (not the latest trunk). Off the bat, does anything look strange to you? It seems from the exception, that the workflow met is definitely not being properly passed to the batch stub.
I know a lot of updates have been made to WM lately, so I tried to build the latest trunk version of 0.4-SNAPSHOT; however, it currently fails the 'TestXmlRpcWorkflowManagerClient' test with:
java.text.ParseException: An exception occurred because the input date/time string was not at least 24 characters in length.
at org.apache.oodt.commons.util.DateConvert.isoParse(DateConvert.java:165)
at org.apache.oodt.cas.workflow.structs.WorkflowInstance.setCurrentTaskStartDateTimeIsoStr(WorkflowInstance.java:339)
I'm going to look more into it and investigate, but maybe I will try deploying with 0.3 OODT and see if the problem persists.
Thanks!
rishi
On Oct 24, 2011, at 9:07 PM, Mattmann, Chris A (388J) wrote:
> Hi Rishi,
>
> The metadata context should be persisted, based on updates made during the tasks themselves. If you have a look at
> TaskJob [1], you'll see the last step in the execute(JobInput in) method is:
>
> {code}
> // now we have to update the workflow manager with the metadata
> // that may have been updated
> updateMetadata(taskInput.getDynMetadata());
> {code}
>
> which should call back to the parent workflow manager and update the workflow instance
> with the current metadata. If you're not seeing this it's possible that your XmlRpcBatchStub
> on your executing node cannot see back to the parent WM running on host:port.
>
> Is this the case?
>
> Cheers,
> Chris
>
> On Oct 24, 2011, at 4:52 PM, Verma, Rishi (388J) wrote:
>
>> Hi all,
>>
>> I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
>>
>> I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
>>
>> By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
>>
>> I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
>>
>> Thanks!
>> rishi
>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Sharing workflow task metadata when using a resource manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Rishi,
The metadata context should be persisted, based on updates made during the tasks themselves. If you have a look at
TaskJob [1], you'll see the last step in the execute(JobInput in) method is:
{code}
// now we have to update the workflow manager with the metadata
// that may have been updated
updateMetadata(taskInput.getDynMetadata());
{code}
which should call back to the parent workflow manager and update the workflow instance
with the current metadata. If you're not seeing this it's possible that your XmlRpcBatchStub
on your executing node cannot see back to the parent WM running on host:port.
Is this the case?
Cheers,
Chris
On Oct 24, 2011, at 4:52 PM, Verma, Rishi (388J) wrote:
> Hi all,
>
> I'm trying to pass generated metadata from a workflow task to another workflow task (both in the same workflow) when using a resource manager to execute workflow task jobs.
>
> I've done this before successfully when workflow tasks within a given workflow are run locally (by the workflow manager itself) but when I point workflow manager to have tasks execute through a resource manager, my generated metadata does not seem to transfer from one task to the next.
>
> By "generated" metadata, I mean metadata that is added within the "run" method of an implemented WorkflowTaskInstance. It's worth noting though, that metadata passed into the initial XmlRpc call of a workflow task seems to be passed to all subsequent tasks just fine. Just not generated metadata - which passes only when not using a resource manager.
>
> I'm trying to ascertain if this issue is a bug or not. To find out, could someone elaborate a bit on which resmgr (or other) classes would include code that actually shows metadata for a job being passed through a remotely running job? I've been trying to find such code within the codebase but have not come across it yet.
>
> Thanks!
> rishi
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Sharing workflow task metadata when using a resource manager
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Thanks for the heads up Brian,
Actually, I am trying to pass workflow met from a WorkflowTaskInstance to a PGETaskInstance, but I will keep this in mind for future cases!
thanks,
rishi
On Oct 24, 2011, at 6:17 PM, Brian Foster wrote:
> if you are using CAS-PGE then in your pgeconfig.xml file you can specify on a any custom metadata element the attribute workflowMet='true' and it will pass through to all the following tasks in the workflow... for example: <metadata key='filename' val='data.dat' workflowMet='true'/>
>
> -Brian
>
> On Oct 24, 2011, at 5:45 PM, "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov> wrote:
>
>> Hey Sheryl,
>>
>> Thanks for the response!
>>
>> Yeah, the workflow-manager log actually already spits out a lot of workflow task instance info whenever it submits a task to a resource manager node.
>> It shows (for each task invoked):
>> * 'task.intance.class' name
>> * 'task.config' parameters
>> * 'task.metadata' contents
>>
>> I did some tests, and concluded (from viewing task.metadata) that all task metadata fed into the first task of my workflow (ie. when passing in met via: wmgr-client ... --key name value...) showed up in the 'task.metadata' for all tasks in my workflow. However, if my first task added new generated metadata, it would not show up in the 'task.metadata' of subsequent tasks being called. The latter issue only occurs when using a resource manager for task execution.
>>
>> Thanks for the tip though, I will keep this in mind while trying out some new strategies.
>>
>> rishi
>>
>> On Oct 24, 2011, at 5:18 PM, Sheryl John wrote:
>>
>>> Hi Rishi,
>>>
>>> Would getting workflowInstance metadata help you debug this problem ( using
>>> command-line option for the wmgr-client)? This will not give you the job
>>> metadata of course, but it gives out the job ids.
>>> And you could probably use the job id for checking out the metadata for
>>> that respective job. I'm not sure if there is code for this.
>>>
>>> On Mon, Oct 24, 2011 at 4:52 PM, Verma, Rishi (388J) <
>>> Rishi.Verma@jpl.nasa.gov> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I'm trying to pass generated metadata from a workflow task to another
>>>> workflow task (both in the same workflow) when using a resource manager to
>>>> execute workflow task jobs.
>>>>
>>>> I've done this before successfully when workflow tasks within a given
>>>> workflow are run locally (by the workflow manager itself) but when I point
>>>> workflow manager to have tasks execute through a resource manager, my
>>>> generated metadata does not seem to transfer from one task to the next.
>>>>
>>>> By "generated" metadata, I mean metadata that is added within the "run"
>>>> method of an implemented WorkflowTaskInstance. It's worth noting though,
>>>> that metadata passed into the initial XmlRpc call of a workflow task seems
>>>> to be passed to all subsequent tasks just fine. Just not generated metadata
>>>> - which passes only when not using a resource manager.
>>>>
>>>> I'm trying to ascertain if this issue is a bug or not. To find out, could
>>>> someone elaborate a bit on which resmgr (or other) classes would include
>>>> code that actually shows metadata for a job being passed through a remotely
>>>> running job? I've been trying to find such code within the codebase but have
>>>> not come across it yet.
>>>>
>>>> Thanks!
>>>> rishi
>>>>
>>>>
>>>
>>>
>>> --
>>> -Sheryl
>>
Re: Sharing workflow task metadata when using a resource manager
Posted by Brian Foster <ho...@me.com>.
if you are using CAS-PGE then in your pgeconfig.xml file you can specify on a any custom metadata element the attribute workflowMet='true' and it will pass through to all the following tasks in the workflow... for example: <metadata key='filename' val='data.dat' workflowMet='true'/>
-Brian
On Oct 24, 2011, at 5:45 PM, "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> Thanks for the response!
>
> Yeah, the workflow-manager log actually already spits out a lot of workflow task instance info whenever it submits a task to a resource manager node.
> It shows (for each task invoked):
> * 'task.intance.class' name
> * 'task.config' parameters
> * 'task.metadata' contents
>
> I did some tests, and concluded (from viewing task.metadata) that all task metadata fed into the first task of my workflow (ie. when passing in met via: wmgr-client ... --key name value...) showed up in the 'task.metadata' for all tasks in my workflow. However, if my first task added new generated metadata, it would not show up in the 'task.metadata' of subsequent tasks being called. The latter issue only occurs when using a resource manager for task execution.
>
> Thanks for the tip though, I will keep this in mind while trying out some new strategies.
>
> rishi
>
> On Oct 24, 2011, at 5:18 PM, Sheryl John wrote:
>
>> Hi Rishi,
>>
>> Would getting workflowInstance metadata help you debug this problem ( using
>> command-line option for the wmgr-client)? This will not give you the job
>> metadata of course, but it gives out the job ids.
>> And you could probably use the job id for checking out the metadata for
>> that respective job. I'm not sure if there is code for this.
>>
>> On Mon, Oct 24, 2011 at 4:52 PM, Verma, Rishi (388J) <
>> Rishi.Verma@jpl.nasa.gov> wrote:
>>
>>> Hi all,
>>>
>>> I'm trying to pass generated metadata from a workflow task to another
>>> workflow task (both in the same workflow) when using a resource manager to
>>> execute workflow task jobs.
>>>
>>> I've done this before successfully when workflow tasks within a given
>>> workflow are run locally (by the workflow manager itself) but when I point
>>> workflow manager to have tasks execute through a resource manager, my
>>> generated metadata does not seem to transfer from one task to the next.
>>>
>>> By "generated" metadata, I mean metadata that is added within the "run"
>>> method of an implemented WorkflowTaskInstance. It's worth noting though,
>>> that metadata passed into the initial XmlRpc call of a workflow task seems
>>> to be passed to all subsequent tasks just fine. Just not generated metadata
>>> - which passes only when not using a resource manager.
>>>
>>> I'm trying to ascertain if this issue is a bug or not. To find out, could
>>> someone elaborate a bit on which resmgr (or other) classes would include
>>> code that actually shows metadata for a job being passed through a remotely
>>> running job? I've been trying to find such code within the codebase but have
>>> not come across it yet.
>>>
>>> Thanks!
>>> rishi
>>>
>>>
>>
>>
>> --
>> -Sheryl
>
Re: Sharing workflow task metadata when using a resource manager
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hey Sheryl,
Thanks for the response!
Yeah, the workflow-manager log actually already spits out a lot of workflow task instance info whenever it submits a task to a resource manager node.
It shows (for each task invoked):
* 'task.intance.class' name
* 'task.config' parameters
* 'task.metadata' contents
I did some tests, and concluded (from viewing task.metadata) that all task metadata fed into the first task of my workflow (ie. when passing in met via: wmgr-client ... --key name value...) showed up in the 'task.metadata' for all tasks in my workflow. However, if my first task added new generated metadata, it would not show up in the 'task.metadata' of subsequent tasks being called. The latter issue only occurs when using a resource manager for task execution.
Thanks for the tip though, I will keep this in mind while trying out some new strategies.
rishi
On Oct 24, 2011, at 5:18 PM, Sheryl John wrote:
> Hi Rishi,
>
> Would getting workflowInstance metadata help you debug this problem ( using
> command-line option for the wmgr-client)? This will not give you the job
> metadata of course, but it gives out the job ids.
> And you could probably use the job id for checking out the metadata for
> that respective job. I'm not sure if there is code for this.
>
> On Mon, Oct 24, 2011 at 4:52 PM, Verma, Rishi (388J) <
> Rishi.Verma@jpl.nasa.gov> wrote:
>
>> Hi all,
>>
>> I'm trying to pass generated metadata from a workflow task to another
>> workflow task (both in the same workflow) when using a resource manager to
>> execute workflow task jobs.
>>
>> I've done this before successfully when workflow tasks within a given
>> workflow are run locally (by the workflow manager itself) but when I point
>> workflow manager to have tasks execute through a resource manager, my
>> generated metadata does not seem to transfer from one task to the next.
>>
>> By "generated" metadata, I mean metadata that is added within the "run"
>> method of an implemented WorkflowTaskInstance. It's worth noting though,
>> that metadata passed into the initial XmlRpc call of a workflow task seems
>> to be passed to all subsequent tasks just fine. Just not generated metadata
>> - which passes only when not using a resource manager.
>>
>> I'm trying to ascertain if this issue is a bug or not. To find out, could
>> someone elaborate a bit on which resmgr (or other) classes would include
>> code that actually shows metadata for a job being passed through a remotely
>> running job? I've been trying to find such code within the codebase but have
>> not come across it yet.
>>
>> Thanks!
>> rishi
>>
>>
>
>
> --
> -Sheryl
Re: Sharing workflow task metadata when using a resource manager
Posted by Sheryl John <sh...@gmail.com>.
Hi Rishi,
Would getting workflowInstance metadata help you debug this problem ( using
command-line option for the wmgr-client)? This will not give you the job
metadata of course, but it gives out the job ids.
And you could probably use the job id for checking out the metadata for
that respective job. I'm not sure if there is code for this.
On Mon, Oct 24, 2011 at 4:52 PM, Verma, Rishi (388J) <
Rishi.Verma@jpl.nasa.gov> wrote:
> Hi all,
>
> I'm trying to pass generated metadata from a workflow task to another
> workflow task (both in the same workflow) when using a resource manager to
> execute workflow task jobs.
>
> I've done this before successfully when workflow tasks within a given
> workflow are run locally (by the workflow manager itself) but when I point
> workflow manager to have tasks execute through a resource manager, my
> generated metadata does not seem to transfer from one task to the next.
>
> By "generated" metadata, I mean metadata that is added within the "run"
> method of an implemented WorkflowTaskInstance. It's worth noting though,
> that metadata passed into the initial XmlRpc call of a workflow task seems
> to be passed to all subsequent tasks just fine. Just not generated metadata
> - which passes only when not using a resource manager.
>
> I'm trying to ascertain if this issue is a bug or not. To find out, could
> someone elaborate a bit on which resmgr (or other) classes would include
> code that actually shows metadata for a job being passed through a remotely
> running job? I've been trying to find such code within the codebase but have
> not come across it yet.
>
> Thanks!
> rishi
>
>
--
-Sheryl