You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> on 2012/02/07 22:51:12 UTC
CAS PGE Crawler Question
Hi all,
I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
<task id="urn:my:Task" name="Task"
...
<property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
<property name="PCS_ActionRepoFile" value="crawler-config.xml" />
</configuration>
</task>
My crawler-config.xml has the following:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
<import resource="./action-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
</beans>
Within my action-beans.xml, I added the post ingest success action:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
<!-- Crawler Actions -->
<bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
<property name="description" value="Kickoff Another Workflow Event"/>
<property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
<property name="phases">
<list>
<value type="java.lang.String">postIngestSuccess</value>
</list>
</property>
</bean>
Not sure if I'm missing anything else.
Thanks in advance,
Mike
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Thanks. Have you tried the below? Looks good to me. Let me know if you have a specific error
or issue and then I'd be happy to help.
Cheers,
Chris
On Feb 7, 2012, at 1:51 PM, Cayanan, Michael D (388J) wrote:
> Hi all,
>
> I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
>
> I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
>
> <task id="urn:my:Task" name="Task"
> ...
> <property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
> <property name="PCS_ActionRepoFile" value="crawler-config.xml" />
> </configuration>
> </task>
>
> My crawler-config.xml has the following:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:p="http://www.springframework.org/schema/p"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
> <import resource="./action-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
> </beans>
>
> Within my action-beans.xml, I added the post ingest success action:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
>
> <!-- Crawler Actions -->
>
> <bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
> <property name="description" value="Kickoff Another Workflow Event"/>
> <property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
> <property name="phases">
> <list>
> <value type="java.lang.String">postIngestSuccess</value>
> </list>
> </property>
> </bean>
>
> Not sure if I'm missing anything else.
>
> Thanks in advance,
> Mike
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Thanks. Have you tried the below? Looks good to me. Let me know if you have a specific error
or issue and then I'd be happy to help.
Cheers,
Chris
On Feb 7, 2012, at 1:51 PM, Cayanan, Michael D (388J) wrote:
> Hi all,
>
> I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
>
> I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
>
> <task id="urn:my:Task" name="Task"
> ...
> <property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
> <property name="PCS_ActionRepoFile" value="crawler-config.xml" />
> </configuration>
> </task>
>
> My crawler-config.xml has the following:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:p="http://www.springframework.org/schema/p"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
> <import resource="./action-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
> </beans>
>
> Within my action-beans.xml, I added the post ingest success action:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
>
> <!-- Crawler Actions -->
>
> <bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
> <property name="description" value="Kickoff Another Workflow Event"/>
> <property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
> <property name="phases">
> <list>
> <value type="java.lang.String">postIngestSuccess</value>
> </list>
> </property>
> </bean>
>
> Not sure if I'm missing anything else.
>
> Thanks in advance,
> Mike
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Mike, you rock!
Cheers,
Chris
On Feb 9, 2012, at 7:21 AM, Cayanan, Michael D (388J) wrote:
> No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to
> specify multiple crawler actions.
>
> Thanks to you and Brian for your help on this,
> Mike
>
> On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Great job, Mike! Would you be willing to write up a quick wiki FAQ on
>> this,
>> here?
>>
>> https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document
>> ation
>>
>> Cheers,
>> Chris
>>
>> On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
>>
>>> Hey Brian and Chris,
>>>
>>> So, I first re-tried it using Chris' suggestion of using spaces and I
>>> still got an error. Here is what I'm seeing in the log:
>>>
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> INFO: Executing StdProductCrawler in productPath:
>>> [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> WARNING: Failed while attempting to ingest products while crawling
>>> directory
>>> '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all
>>> products may not have been ingested) : No bean named
>>> 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No
>>> bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getB
>>> eanDefinition(DefaultListableBeanFactory.java:360)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe
>>> anDefinition(AbstractBeanFactory.java:914)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:241)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:164)
>>> at
>>> org.springframework.context.support.AbstractApplicationContext.getBean(Ab
>>> stractApplicationContext.java:761)
>>> at
>>> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac
>>> tory(CrawlerActionRepo.java:68)
>>> at
>>> org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.
>>> java:405)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja
>>> va:349)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>>> at
>>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>> at
>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru
>>> n(XmlRpcBatchStub.java:229)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>>
>>> I then modified my tasks.xml file where I specified the
>>> 'PCS_ActionsIds' property with the action IDs:
>>>
>>> <property name="PCS_ActionsIds"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>>
>>> and added a line in the PgeConfig.xml as Brian suggested:
>>>
>>> <customMetadata>
>>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>>
>>> and still got an error in the log as shown above.
>>>
>>> So, I ended up re-naming the property name in the tasks.xml to some
>>> other name (so as to not reference itself essentially) and that did the
>>> trick!
>>>
>>> tasks.xml:
>>>
>>> <property name="Actions"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>>
>>> PgeConfig.xml:
>>>
>>> <customMetadata>
>>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>>
>>>
>>> Also, I found out that you can't have spaces in between the commas like
>>> so,
>>>
>>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,
>>> MoveMetadataToBackupDir" />
>>>
>>>
>>> Thanks for all your help guys!
>>>
>>>
>>> -Mike
>>>
>>>
>>> From: holenoter <ho...@mac.com>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Subject: Re: CAS PGE Crawler Question
>>>
>>> workflow manager properties (static metadata) are passed to CAS-PGE via
>>> WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE
>>> doesn't re-parse out these metadata value because you have to allow for
>>> workflow metadata to contain values with commas without always being
>>> split out by CAS-PGE.. so you have to explicitly repoint the metadata
>>> field at itself so it becomes custom (or local) metadata for CAS-PGE and
>>> then the value will be parsed on comma... context: CAS-PGE initially did
>>> the opposite (i.e. always re-parse workflow metadata into multi-values)
>>> but chris later changed this to the way it is now because this broke OCO
>>> since OCO had static workflow metadata which contained metadata with
>>> commas that was not supposed to be parsed into multi-values.
>>>
>>> -brian
>>>
>>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)"
>>> <ch...@jpl.nasa.gov> wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> Space should work in the property value below. Can you show me
>>>> the log output from CAS-PGE in the workflow log file (or the resource
>>>> manager
>>>> batch stubs?) It should show you what Actions are being executed
>>>> and that can help to debug.
>>>>
>>>> Thanks!
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>>
>>>>> If I wanted to specify more than one PCS_ActionsIds values, how
>>>> would I do
>>>>> this?
>>>>>
>>>>> I've tried the following with no success:
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1, Action2" />
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1,Action2" />
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1 Action2" />
>>>>>
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>>
>>>>> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hi Mike,
>>>>>>
>>>>>> Glad to hear!
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Disregard my previous message. Turns out I had a typo. I had to set
>>>>>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>>>>>
>>>>>>> -Mike
>>>>>>>
>>>>>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>>>>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>>>>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>>>>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>>>>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>>>>>> Subject: CAS PGE Crawler Question
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I would like CAS PGE to run a post ingest success action that I
>>>> created
>>>>>>> upon a successful ingestion of the output file(s). How do you
>>>> configure
>>>>>>> CAS PGE for this type of scenario?
>>>>>>>
>>>>>>> I have a tasks.xml file in the workflow, where I define my post
>>>> ingest
>>>>>>> action under the 'PCS_ActionIds' property:
>>>>>>>
>>>>>>> <task id="urn:my:Task" name="Task"
>>>>>>> ...
>>>>>>> <property name="PCS_ActionIds"
>>>>>>> value="MyPostIngestSuccessAction" />
>>>>>>> <property name="PCS_ActionRepoFile"
>>>>>>> value="crawler-config.xml" />
>>>>>>> </configuration>
>>>>>>> </task>
>>>>>>>
>>>>>>> My crawler-config.xml has the following:
>>>>>>>
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xmlns:p="http://www.springframework.org/schema/p"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>>
>>>>>>> <import
>>>>>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>>>>>> <import resource="./action-beans.xml"/>
>>>>>>> <import
>>>>>>>
>>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b
>>>> e
>>>>>>> ans.xml"/>
>>>>>>> <import
>>>>>>>
>>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be
>>>> a
>>>>>>> ns.xml"/>
>>>>>>> </beans>
>>>>>>>
>>>>>>> Within my action-beans.xml, I added the post ingest success action:
>>>>>>>
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>>
>>>>>>> <bean
>>>>>>>
>>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess
>>>> o
>>>>>>> r"/>
>>>>>>>
>>>>>>> <!-- Crawler Actions -->
>>>>>>>
>>>>>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>>>>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>>>>>> <property name="description" value="Kickoff Another Workflow
>>>>>>> Event"/>
>>>>>>> <property name="executeCommand" value="sh
>>>>>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>>>>>> <property name="phases">
>>>>>>> <list>
>>>>>>> <value type="java.lang.String">postIngestSuccess</value>
>>>>>>> </list>
>>>>>>> </property>
>>>>>>> </bean>
>>>>>>>
>>>>>>> Not sure if I'm missing anything else.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Mike
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Mike, you rock!
Cheers,
Chris
On Feb 9, 2012, at 7:21 AM, Cayanan, Michael D (388J) wrote:
> No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to
> specify multiple crawler actions.
>
> Thanks to you and Brian for your help on this,
> Mike
>
> On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Great job, Mike! Would you be willing to write up a quick wiki FAQ on
>> this,
>> here?
>>
>> https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document
>> ation
>>
>> Cheers,
>> Chris
>>
>> On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
>>
>>> Hey Brian and Chris,
>>>
>>> So, I first re-tried it using Chris' suggestion of using spaces and I
>>> still got an error. Here is what I'm seeing in the log:
>>>
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> INFO: Executing StdProductCrawler in productPath:
>>> [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>> runIngestCrawler
>>> WARNING: Failed while attempting to ingest products while crawling
>>> directory
>>> '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all
>>> products may not have been ingested) : No bean named
>>> 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No
>>> bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>>> at
>>> org.springframework.beans.factory.support.DefaultListableBeanFactory.getB
>>> eanDefinition(DefaultListableBeanFactory.java:360)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe
>>> anDefinition(AbstractBeanFactory.java:914)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:241)
>>> at
>>> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>> tractBeanFactory.java:164)
>>> at
>>> org.springframework.context.support.AbstractApplicationContext.getBean(Ab
>>> stractApplicationContext.java:761)
>>> at
>>> org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac
>>> tory(CrawlerActionRepo.java:68)
>>> at
>>> org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.
>>> java:405)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja
>>> va:349)
>>> at
>>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>>> at
>>> org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>>> at
>>> org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru
>>> n(XmlRpcBatchStub.java:229)
>>> at java.lang.Thread.run(Thread.java:680)
>>>
>>>
>>> I then modified my tasks.xml file where I specified the
>>> 'PCS_ActionsIds' property with the action IDs:
>>>
>>> <property name="PCS_ActionsIds"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>>
>>> and added a line in the PgeConfig.xml as Brian suggested:
>>>
>>> <customMetadata>
>>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>>
>>> and still got an error in the log as shown above.
>>>
>>> So, I ended up re-naming the property name in the tasks.xml to some
>>> other name (so as to not reference itself essentially) and that did the
>>> trick!
>>>
>>> tasks.xml:
>>>
>>> <property name="Actions"
>>> value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>>
>>> PgeConfig.xml:
>>>
>>> <customMetadata>
>>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>>
>>>
>>> Also, I found out that you can't have spaces in between the commas like
>>> so,
>>>
>>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,
>>> MoveMetadataToBackupDir" />
>>>
>>>
>>> Thanks for all your help guys!
>>>
>>>
>>> -Mike
>>>
>>>
>>> From: holenoter <ho...@mac.com>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Subject: Re: CAS PGE Crawler Question
>>>
>>> workflow manager properties (static metadata) are passed to CAS-PGE via
>>> WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE
>>> doesn't re-parse out these metadata value because you have to allow for
>>> workflow metadata to contain values with commas without always being
>>> split out by CAS-PGE.. so you have to explicitly repoint the metadata
>>> field at itself so it becomes custom (or local) metadata for CAS-PGE and
>>> then the value will be parsed on comma... context: CAS-PGE initially did
>>> the opposite (i.e. always re-parse workflow metadata into multi-values)
>>> but chris later changed this to the way it is now because this broke OCO
>>> since OCO had static workflow metadata which contained metadata with
>>> commas that was not supposed to be parsed into multi-values.
>>>
>>> -brian
>>>
>>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)"
>>> <ch...@jpl.nasa.gov> wrote:
>>>
>>>> Hi Mike,
>>>>
>>>> Space should work in the property value below. Can you show me
>>>> the log output from CAS-PGE in the workflow log file (or the resource
>>>> manager
>>>> batch stubs?) It should show you what Actions are being executed
>>>> and that can help to debug.
>>>>
>>>> Thanks!
>>>>
>>>> Cheers,
>>>> Chris
>>>>
>>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>>
>>>>> If I wanted to specify more than one PCS_ActionsIds values, how
>>>> would I do
>>>>> this?
>>>>>
>>>>> I've tried the following with no success:
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1, Action2" />
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1,Action2" />
>>>>>
>>>>>
>>>>> and
>>>>>
>>>>> <property name="PCS_ActionIds" value="Action1 Action2" />
>>>>>
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>>
>>>>> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hi Mike,
>>>>>>
>>>>>> Glad to hear!
>>>>>>
>>>>>> Cheers,
>>>>>> Chris
>>>>>>
>>>>>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Disregard my previous message. Turns out I had a typo. I had to set
>>>>>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>>>>>
>>>>>>> -Mike
>>>>>>>
>>>>>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>>>>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>>>>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>>>>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>>>>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>>>>>> Subject: CAS PGE Crawler Question
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I would like CAS PGE to run a post ingest success action that I
>>>> created
>>>>>>> upon a successful ingestion of the output file(s). How do you
>>>> configure
>>>>>>> CAS PGE for this type of scenario?
>>>>>>>
>>>>>>> I have a tasks.xml file in the workflow, where I define my post
>>>> ingest
>>>>>>> action under the 'PCS_ActionIds' property:
>>>>>>>
>>>>>>> <task id="urn:my:Task" name="Task"
>>>>>>> ...
>>>>>>> <property name="PCS_ActionIds"
>>>>>>> value="MyPostIngestSuccessAction" />
>>>>>>> <property name="PCS_ActionRepoFile"
>>>>>>> value="crawler-config.xml" />
>>>>>>> </configuration>
>>>>>>> </task>
>>>>>>>
>>>>>>> My crawler-config.xml has the following:
>>>>>>>
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xmlns:p="http://www.springframework.org/schema/p"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>>
>>>>>>> <import
>>>>>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>>>>>> <import resource="./action-beans.xml"/>
>>>>>>> <import
>>>>>>>
>>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b
>>>> e
>>>>>>> ans.xml"/>
>>>>>>> <import
>>>>>>>
>>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be
>>>> a
>>>>>>> ns.xml"/>
>>>>>>> </beans>
>>>>>>>
>>>>>>> Within my action-beans.xml, I added the post ingest success action:
>>>>>>>
>>>>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>>>>>
>>>>>>> <bean
>>>>>>>
>>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess
>>>> o
>>>>>>> r"/>
>>>>>>>
>>>>>>> <!-- Crawler Actions -->
>>>>>>>
>>>>>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>>>>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>>>>>> <property name="description" value="Kickoff Another Workflow
>>>>>>> Event"/>
>>>>>>> <property name="executeCommand" value="sh
>>>>>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>>>>>> <property name="phases">
>>>>>>> <list>
>>>>>>> <value type="java.lang.String">postIngestSuccess</value>
>>>>>>> </list>
>>>>>>> </property>
>>>>>>> </bean>
>>>>>>>
>>>>>>> Not sure if I'm missing anything else.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>> Mike
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Chris Mattmann, Ph.D.
>>>>>> Senior Computer Scientist
>>>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>> Office: 171-266B, Mailstop: 171-246
>>>>>> Email: chris.a.mattmann@nasa.gov
>>>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> Adjunct Assistant Professor, Computer Science Department
>>>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>
>>>>>
>>>>
>>>>
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Chris Mattmann, Ph.D.
>>>> Senior Computer Scientist
>>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>> Office: 171-266B, Mailstop: 171-246
>>>> Email: chris.a.mattmann@nasa.gov
>>>> WWW: http://sunset.usc.edu/~mattmann/
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> Adjunct Assistant Professor, Computer Science Department
>>>> University of Southern California, Los Angeles, CA 90089 USA
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to
specify multiple crawler actions.
Thanks to you and Brian for your help on this,
Mike
On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Great job, Mike! Would you be willing to write up a quick wiki FAQ on
>this,
>here?
>
>https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document
>ation
>
>Cheers,
>Chris
>
>On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I
>>still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath:
>>[/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling
>>directory
>>'/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all
>>products may not have been ingested) : No bean named
>>'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No
>>bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at
>>org.springframework.beans.factory.support.DefaultListableBeanFactory.getB
>>eanDefinition(DefaultListableBeanFactory.java:360)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe
>>anDefinition(AbstractBeanFactory.java:914)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>tractBeanFactory.java:241)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>tractBeanFactory.java:164)
>> at
>>org.springframework.context.support.AbstractApplicationContext.getBean(Ab
>>stractApplicationContext.java:761)
>> at
>>org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac
>>tory(CrawlerActionRepo.java:68)
>> at
>>org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.
>>java:405)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja
>>va:349)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at
>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at
>>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru
>>n(XmlRpcBatchStub.java:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the
>>'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds"
>>value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some
>>other name (so as to not reference itself essentially) and that did the
>>trick!
>>
>> tasks.xml:
>>
>> <property name="Actions"
>>value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like
>>so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,
>>MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via
>>WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE
>>doesn't re-parse out these metadata value because you have to allow for
>>workflow metadata to contain values with commas without always being
>>split out by CAS-PGE.. so you have to explicitly repoint the metadata
>>field at itself so it becomes custom (or local) metadata for CAS-PGE and
>>then the value will be parsed on comma... context: CAS-PGE initially did
>>the opposite (i.e. always re-parse workflow metadata into multi-values)
>>but chris later changed this to the way it is now because this broke OCO
>>since OCO had static workflow metadata which contained metadata with
>>commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)"
>><ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource
>>>manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how
>>>would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I
>>>created
>>> >>> upon a successful ingestion of the output file(s). How do you
>>>configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post
>>>ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>>
>>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b
>>>e
>>> >>> ans.xml"/>
>>> >>> <import
>>> >>>
>>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be
>>>a
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>>
>>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess
>>>o
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
No problemo Mr Mattmann. I've added a FAQ section to the wiki on how to
specify multiple crawler actions.
Thanks to you and Brian for your help on this,
Mike
On 2/8/12 7:44 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Great job, Mike! Would you be willing to write up a quick wiki FAQ on
>this,
>here?
>
>https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Document
>ation
>
>Cheers,
>Chris
>
>On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I
>>still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath:
>>[/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance
>>runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling
>>directory
>>'/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all
>>products may not have been ingested) : No bean named
>>'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No
>>bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at
>>org.springframework.beans.factory.support.DefaultListableBeanFactory.getB
>>eanDefinition(DefaultListableBeanFactory.java:360)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBe
>>anDefinition(AbstractBeanFactory.java:914)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>tractBeanFactory.java:241)
>> at
>>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(Abs
>>tractBeanFactory.java:164)
>> at
>>org.springframework.context.support.AbstractApplicationContext.getBean(Ab
>>stractApplicationContext.java:761)
>> at
>>org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFac
>>tory(CrawlerActionRepo.java:68)
>> at
>>org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.
>>java:405)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.ja
>>va:349)
>> at
>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at
>>org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at
>>org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.ru
>>n(XmlRpcBatchStub.java:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the
>>'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds"
>>value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some
>>other name (so as to not reference itself essentially) and that did the
>>trick!
>>
>> tasks.xml:
>>
>> <property name="Actions"
>>value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like
>>so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,
>>MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via
>>WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE
>>doesn't re-parse out these metadata value because you have to allow for
>>workflow metadata to contain values with commas without always being
>>split out by CAS-PGE.. so you have to explicitly repoint the metadata
>>field at itself so it becomes custom (or local) metadata for CAS-PGE and
>>then the value will be parsed on comma... context: CAS-PGE initially did
>>the opposite (i.e. always re-parse workflow metadata into multi-values)
>>but chris later changed this to the way it is now because this broke OCO
>>since OCO had static workflow metadata which contained metadata with
>>commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)"
>><ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource
>>>manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how
>>>would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I
>>>created
>>> >>> upon a successful ingestion of the output file(s). How do you
>>>configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post
>>>ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>>
>>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-b
>>>e
>>> >>> ans.xml"/>
>>> >>> <import
>>> >>>
>>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-be
>>>a
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>>
>>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcess
>>>o
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Great job, Mike! Would you be willing to write up a quick wiki FAQ on this,
here?
https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Documentation
Cheers,
Chris
On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian and Chris,
>
> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
> at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
> at java.lang.Thread.run(Thread.java:680)
>
>
> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>
> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>
> and added a line in the PgeConfig.xml as Brian suggested:
>
> <customMetadata>
> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>
> and still got an error in the log as shown above.
>
> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>
> tasks.xml:
>
> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>
> PgeConfig.xml:
>
> <customMetadata>
> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>
>
> Also, I found out that you can't have spaces in between the commas like so,
>
> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>
>
> Thanks for all your help guys!
>
>
> -Mike
>
>
> From: holenoter <ho...@mac.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Tue, 7 Feb 2012 22:06:27 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: CAS PGE Crawler Question
>
> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>
> -brian
>
> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Space should work in the property value below. Can you show me
>> the log output from CAS-PGE in the workflow log file (or the resource manager
>> batch stubs?) It should show you what Actions are being executed
>> and that can help to debug.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>
>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>> > this?
>> >
>> > I've tried the following with no success:
>> >
>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>> >
>> >
>> > and
>> >
>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>> >
>> >
>> > and
>> >
>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>> >
>> >
>> >
>> > -Mike
>> >
>> >
>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>> > <ch...@jpl.nasa.gov> wrote:
>> >
>> >> Hi Mike,
>> >>
>> >> Glad to hear!
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>> >>>
>> >>> -Mike
>> >>>
>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> >>> Subject: CAS PGE Crawler Question
>> >>>
>> >>> Hi all,
>> >>>
>> >>> I would like CAS PGE to run a post ingest success action that I created
>> >>> upon a successful ingestion of the output file(s). How do you configure
>> >>> CAS PGE for this type of scenario?
>> >>>
>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>> >>> action under the 'PCS_ActionIds' property:
>> >>>
>> >>> <task id="urn:my:Task" name="Task"
>> >>> ...
>> >>> <property name="PCS_ActionIds"
>> >>> value="MyPostIngestSuccessAction" />
>> >>> <property name="PCS_ActionRepoFile"
>> >>> value="crawler-config.xml" />
>> >>> </configuration>
>> >>> </task>
>> >>>
>> >>> My crawler-config.xml has the following:
>> >>>
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xmlns:p="http://www.springframework.org/schema/p"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> >>> <import resource="./action-beans.xml"/>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>> >>> ans.xml"/>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>> >>> ns.xml"/>
>> >>> </beans>
>> >>>
>> >>> Within my action-beans.xml, I added the post ingest success action:
>> >>>
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>>
>> >>> <bean
>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>> >>> r"/>
>> >>>
>> >>> <!-- Crawler Actions -->
>> >>>
>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> >>> <property name="description" value="Kickoff Another Workflow
>> >>> Event"/>
>> >>> <property name="executeCommand" value="sh
>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> >>> <property name="phases">
>> >>> <list>
>> >>> <value type="java.lang.String">postIngestSuccess</value>
>> >>> </list>
>> >>> </property>
>> >>> </bean>
>> >>>
>> >>> Not sure if I'm missing anything else.
>> >>>
>> >>> Thanks in advance,
>> >>> Mike
>> >>>
>> >>
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Great job, Mike! Would you be willing to write up a quick wiki FAQ on this,
here?
https://cwiki.apache.org/confluence/display/OODT/CAS-PGE+Help+and+Documentation
Cheers,
Chris
On Feb 8, 2012, at 8:34 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian and Chris,
>
> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
> at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
> at java.lang.Thread.run(Thread.java:680)
>
>
> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>
> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>
> and added a line in the PgeConfig.xml as Brian suggested:
>
> <customMetadata>
> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>
> and still got an error in the log as shown above.
>
> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>
> tasks.xml:
>
> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>
> PgeConfig.xml:
>
> <customMetadata>
> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>
>
> Also, I found out that you can't have spaces in between the commas like so,
>
> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>
>
> Thanks for all your help guys!
>
>
> -Mike
>
>
> From: holenoter <ho...@mac.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Tue, 7 Feb 2012 22:06:27 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: CAS PGE Crawler Question
>
> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>
> -brian
>
> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Space should work in the property value below. Can you show me
>> the log output from CAS-PGE in the workflow log file (or the resource manager
>> batch stubs?) It should show you what Actions are being executed
>> and that can help to debug.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>
>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>> > this?
>> >
>> > I've tried the following with no success:
>> >
>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>> >
>> >
>> > and
>> >
>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>> >
>> >
>> > and
>> >
>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>> >
>> >
>> >
>> > -Mike
>> >
>> >
>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>> > <ch...@jpl.nasa.gov> wrote:
>> >
>> >> Hi Mike,
>> >>
>> >> Glad to hear!
>> >>
>> >> Cheers,
>> >> Chris
>> >>
>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>> >>>
>> >>> -Mike
>> >>>
>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> >>> Subject: CAS PGE Crawler Question
>> >>>
>> >>> Hi all,
>> >>>
>> >>> I would like CAS PGE to run a post ingest success action that I created
>> >>> upon a successful ingestion of the output file(s). How do you configure
>> >>> CAS PGE for this type of scenario?
>> >>>
>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>> >>> action under the 'PCS_ActionIds' property:
>> >>>
>> >>> <task id="urn:my:Task" name="Task"
>> >>> ...
>> >>> <property name="PCS_ActionIds"
>> >>> value="MyPostIngestSuccessAction" />
>> >>> <property name="PCS_ActionRepoFile"
>> >>> value="crawler-config.xml" />
>> >>> </configuration>
>> >>> </task>
>> >>>
>> >>> My crawler-config.xml has the following:
>> >>>
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xmlns:p="http://www.springframework.org/schema/p"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> >>> <import resource="./action-beans.xml"/>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>> >>> ans.xml"/>
>> >>> <import
>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>> >>> ns.xml"/>
>> >>> </beans>
>> >>>
>> >>> Within my action-beans.xml, I added the post ingest success action:
>> >>>
>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>>
>> >>> <bean
>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>> >>> r"/>
>> >>>
>> >>> <!-- Crawler Actions -->
>> >>>
>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> >>> <property name="description" value="Kickoff Another Workflow
>> >>> Event"/>
>> >>> <property name="executeCommand" value="sh
>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> >>> <property name="phases">
>> >>> <list>
>> >>> <value type="java.lang.String">postIngestSuccess</value>
>> >>> </list>
>> >>> </property>
>> >>> </bean>
>> >>>
>> >>> Not sure if I'm missing anything else.
>> >>>
>> >>> Thanks in advance,
>> >>> Mike
>> >>>
>> >>
>> >>
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Chris Mattmann, Ph.D.
>> >> Senior Computer Scientist
>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >> Office: 171-266B, Mailstop: 171-246
>> >> Email: chris.a.mattmann@nasa.gov
>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >> Adjunct Assistant Professor, Computer Science Department
>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>
>> >
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, let's chat some more about it.
I'll take a look maybe later in the week thanks dude...
Cheers,
Chris
On Feb 8, 2012, at 10:27 PM, holenoter wrote:
>
> hey chris,
>
> this particular thing is a one liner... however, if there are current workflows using trunk CAS-PGE they may break due to the change... probably should talk more about maybe making this Java Property configurable or something
>
> -brian
>
> On Feb 08, 2012, at 06:54 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Thanks Bfost.
>>
>> How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
>>
>> I'll take a look at the end of this week, but won't have time before then, as I'm working
>> on Wengine porting, and some cas curator stuff that folks just reported.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>> On Feb 8, 2012, at 9:20 AM, holenoter wrote:
>>
>> > hey mike,
>> >
>> > forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>> >
>> > -brian
>> >
>> > On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >
>> >> Hey Brian and Chris,
>> >>
>> >> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>> >>
>> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> >> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> >> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> >> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> >> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> >> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> >> at org.springframework.beans.factorysupport.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
>> >> at org.springframework.beans.factory.supportAbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> >> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> >> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> >> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> >> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> >> at org.apache.oodtcas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>> >> at java.lang.Thread.run(Thread.java:680)
>> >>
>> >>
>> >> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>> >>
>> >> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>> >>
>> >> and added a line in the PgeConfig.xml as Brian suggested:
>> >>
>> >> <customMetadata>
>> >> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>> >>
>> >> and still got an error in the log as shown above.
>> >>
>> >> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>> >>
>> >> tasks.xml:
>> >>
>> >> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>> >>
>> >> PgeConfig.xml:
>> >>
>> >> <customMetadata>
>> >> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>> >>
>> >>
>> >> Also, I found out that you can't have spaces in between the commas like so,
>> >>
>> >> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>> >>
>> >>
>> >> Thanks for all your help guys!
>> >>
>> >>
>> >> -Mike
>> >>
>> >>
>> >> From: holenoter <ho...@mac.com>
>> >> Reply-To: "user@oodtapache.org" <us...@oodt.apache.org>
>> >> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> >> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >> Subject: Re: CAS PGE Crawler Question
>> >>
>> >> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>> >>
>> >> -brian
>> >>
>> >> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>> >>
>> >>> Hi Mike,
>> >>>
>> >>> Space should work in the property value below. Can you show me
>> >>> the log output from CAS-PGE in the workflow log file (or the resource manager
>> >>> batch stubs?) It should show you what Actions are being executed
>> >>> and that can help to debug.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> Cheers,
>> >>> Chris
>> >>>
>> >>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>> >>>
>> >>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>> >>> > this?
>> >>> >
>> >>> > I've tried the following with no success:
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>> >>> >
>> >>> >
>> >>> > and
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>> >>> >
>> >>> >
>> >>> > and
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>> >>> >
>> >>> >
>> >>> >
>> >>> > -Mike
>> >>> >
>> >>> >
>> >>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>> >>> > <ch...@jpl.nasa.gov> wrote:
>> >>> >
>> >>> >> Hi Mike,
>> >>> >>
>> >>> >> Glad to hear!
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Chris
>> >>> >>
>> >>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>> >>> >>
>> >>> >>> Hi all,
>> >>> >>>
>> >>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>> >>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>> >>> >>>
>> >>> >>> -Mike
>> >>> >>>
>> >>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> >>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> >>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> >>> >>> Subject: CAS PGE Crawler Question
>> >>> >>>
>> >>> >>> Hi all,
>> >>> >>>
>> >>> >>> I would like CAS PGE to run a post ingest success action that I created
>> >>> >>> upon a successful ingestion of the output file(s). How do you configure
>> >>> >>> CAS PGE for this type of scenario?
>> >>> >>>
>> >>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>> >>> >>> action under the 'PCS_ActionIds' property:
>> >>> >>>
>> >>> >>> <task id="urn:my:Task" name="Task"
>> >>> >>> ...
>> >>> >>> <property name="PCS_ActionIds"
>> >>> >>> value="MyPostIngestSuccessAction" />
>> >>> >>> <property name="PCS_ActionRepoFile"
>> >>> >>> value="crawler-config.xml" />
>> >>> >>> </configuration>
>> >>> >>> </task>
>> >>> >>>
>> >>> >>> My crawler-config.xml has the following:
>> >>> >>>
>> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> >>> xmlns:p="http://www.springframework.org/schema/p"
>> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> >>>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> >>> >>> <import resource="./action-beans.xml"/>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>> >>> >>> ansxml"/>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>> >>> >>> ns.xml"/>
>> >>> >>> </beans>
>> >>> >>>
>> >>> >>> Within my action-beans.xml, I added the post ingest success action:
>> >>> >>>
>> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> >>>
>> >>> >>> <bean
>> >>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>> >>> >>> r"/>
>> >>> >>>
>> >>> >>> <!-- Crawler Actions -->
>> >>> >>>
>> >>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>> >>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> >>> >>> <property name="description" value="Kickoff Another Workflow
>> >>> >>> Event"/>
>> >>> >>> <property name="executeCommand" value="sh
>> >>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> >>> >>> <property name="phases">
>> >>> >>> <list>
>> >>> >>> <value type="java.lang.String">postIngestSuccess</value>
>> >>> >>> </list>
>> >>> >>> </property>
>> >>> >>> </bean>
>> >>> >>>
>> >>> >>> Not sure if I'm missing anything else.
>> >>> >>>
>> >>> >>> Thanks in advance,
>> >>> >>> Mike
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >> Chris Mattmann, Ph.D.
>> >>> >> Senior Computer Scientist
>> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> >> Office: 171-266B, Mailstop: 171-246
>> >>> >> Email: chris.a.mattmann@nasa.gov
>> >>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >> Adjunct Assistant Professor, Computer Science Department
>> >>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >>
>> >>> >
>> >>>
>> >>>
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Chris Mattmann, Ph.D
>> >>> Senior Computer Scientist
>> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> Office: 171-266B, Mailstop: 171-246
>> >>> Email: chris.a.mattmann@nasa.gov
>> >>> WWW: http://sunset.usc.edu/~mattmann/
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Adjunct Assistant Professor, Computer Science Department
>> >>> University of Southern California, Los Angeles, CA 90089 USA
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, let's chat some more about it.
I'll take a look maybe later in the week thanks dude...
Cheers,
Chris
On Feb 8, 2012, at 10:27 PM, holenoter wrote:
>
> hey chris,
>
> this particular thing is a one liner... however, if there are current workflows using trunk CAS-PGE they may break due to the change... probably should talk more about maybe making this Java Property configurable or something
>
> -brian
>
> On Feb 08, 2012, at 06:54 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Thanks Bfost.
>>
>> How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
>>
>> I'll take a look at the end of this week, but won't have time before then, as I'm working
>> on Wengine porting, and some cas curator stuff that folks just reported.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>> On Feb 8, 2012, at 9:20 AM, holenoter wrote:
>>
>> > hey mike,
>> >
>> > forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>> >
>> > -brian
>> >
>> > On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >
>> >> Hey Brian and Chris,
>> >>
>> >> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>> >>
>> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> >> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> >> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> >> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> >> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> >> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> >> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> >> at org.springframework.beans.factorysupport.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
>> >> at org.springframework.beans.factory.supportAbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> >> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> >> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> >> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
>> >> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> >> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> >> at org.apache.oodtcas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>> >> at java.lang.Thread.run(Thread.java:680)
>> >>
>> >>
>> >> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>> >>
>> >> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>> >>
>> >> and added a line in the PgeConfig.xml as Brian suggested:
>> >>
>> >> <customMetadata>
>> >> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>> >>
>> >> and still got an error in the log as shown above.
>> >>
>> >> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>> >>
>> >> tasks.xml:
>> >>
>> >> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>> >>
>> >> PgeConfig.xml:
>> >>
>> >> <customMetadata>
>> >> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>> >>
>> >>
>> >> Also, I found out that you can't have spaces in between the commas like so,
>> >>
>> >> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>> >>
>> >>
>> >> Thanks for all your help guys!
>> >>
>> >>
>> >> -Mike
>> >>
>> >>
>> >> From: holenoter <ho...@mac.com>
>> >> Reply-To: "user@oodtapache.org" <us...@oodt.apache.org>
>> >> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> >> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >> Subject: Re: CAS PGE Crawler Question
>> >>
>> >> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>> >>
>> >> -brian
>> >>
>> >> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>> >>
>> >>> Hi Mike,
>> >>>
>> >>> Space should work in the property value below. Can you show me
>> >>> the log output from CAS-PGE in the workflow log file (or the resource manager
>> >>> batch stubs?) It should show you what Actions are being executed
>> >>> and that can help to debug.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> Cheers,
>> >>> Chris
>> >>>
>> >>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>> >>>
>> >>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>> >>> > this?
>> >>> >
>> >>> > I've tried the following with no success:
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>> >>> >
>> >>> >
>> >>> > and
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>> >>> >
>> >>> >
>> >>> > and
>> >>> >
>> >>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>> >>> >
>> >>> >
>> >>> >
>> >>> > -Mike
>> >>> >
>> >>> >
>> >>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>> >>> > <ch...@jpl.nasa.gov> wrote:
>> >>> >
>> >>> >> Hi Mike,
>> >>> >>
>> >>> >> Glad to hear!
>> >>> >>
>> >>> >> Cheers,
>> >>> >> Chris
>> >>> >>
>> >>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>> >>> >>
>> >>> >>> Hi all,
>> >>> >>>
>> >>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>> >>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>> >>> >>>
>> >>> >>> -Mike
>> >>> >>>
>> >>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> >>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> >>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> >>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> >>> >>> Subject: CAS PGE Crawler Question
>> >>> >>>
>> >>> >>> Hi all,
>> >>> >>>
>> >>> >>> I would like CAS PGE to run a post ingest success action that I created
>> >>> >>> upon a successful ingestion of the output file(s). How do you configure
>> >>> >>> CAS PGE for this type of scenario?
>> >>> >>>
>> >>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>> >>> >>> action under the 'PCS_ActionIds' property:
>> >>> >>>
>> >>> >>> <task id="urn:my:Task" name="Task"
>> >>> >>> ...
>> >>> >>> <property name="PCS_ActionIds"
>> >>> >>> value="MyPostIngestSuccessAction" />
>> >>> >>> <property name="PCS_ActionRepoFile"
>> >>> >>> value="crawler-config.xml" />
>> >>> >>> </configuration>
>> >>> >>> </task>
>> >>> >>>
>> >>> >>> My crawler-config.xml has the following:
>> >>> >>>
>> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> >>> xmlns:p="http://www.springframework.org/schema/p"
>> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> >>>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> >>> >>> <import resource="./action-beans.xml"/>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>> >>> >>> ansxml"/>
>> >>> >>> <import
>> >>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>> >>> >>> ns.xml"/>
>> >>> >>> </beans>
>> >>> >>>
>> >>> >>> Within my action-beans.xml, I added the post ingest success action:
>> >>> >>>
>> >>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>> >>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> >>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>> >>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>> >>> >>>
>> >>> >>> <bean
>> >>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>> >>> >>> r"/>
>> >>> >>>
>> >>> >>> <!-- Crawler Actions -->
>> >>> >>>
>> >>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>> >>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> >>> >>> <property name="description" value="Kickoff Another Workflow
>> >>> >>> Event"/>
>> >>> >>> <property name="executeCommand" value="sh
>> >>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> >>> >>> <property name="phases">
>> >>> >>> <list>
>> >>> >>> <value type="java.lang.String">postIngestSuccess</value>
>> >>> >>> </list>
>> >>> >>> </property>
>> >>> >>> </bean>
>> >>> >>>
>> >>> >>> Not sure if I'm missing anything else.
>> >>> >>>
>> >>> >>> Thanks in advance,
>> >>> >>> Mike
>> >>> >>>
>> >>> >>
>> >>> >>
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >> Chris Mattmann, Ph.D.
>> >>> >> Senior Computer Scientist
>> >>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> >> Office: 171-266B, Mailstop: 171-246
>> >>> >> Email: chris.a.mattmann@nasa.gov
>> >>> >> WWW: http://sunset.usc.edu/~mattmann/
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >> Adjunct Assistant Professor, Computer Science Department
>> >>> >> University of Southern California, Los Angeles, CA 90089 USA
>> >>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> >>
>> >>> >
>> >>>
>> >>>
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Chris Mattmann, Ph.D
>> >>> Senior Computer Scientist
>> >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>> Office: 171-266B, Mailstop: 171-246
>> >>> Email: chris.a.mattmann@nasa.gov
>> >>> WWW: http://sunset.usc.edu/~mattmann/
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>> Adjunct Assistant Professor, Computer Science Department
>> >>> University of Southern California, Los Angeles, CA 90089 USA
>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey chris,
this particular thing is a one liner... however, if there are current workflows using trunk CAS-PGE they may break due to the change... probably should talk more about maybe making this Java Property configurable or something
-brian
On Feb 08, 2012, at 06:54 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Thanks Bfost.
How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
I'll take a look at the end of this week, but won't have time before then, as I'm working
on Wengine porting, and some cas curator stuff that folks just reported.
Thanks!
Cheers,
Chris
On Feb 8, 2012, at 9:20 AM, holenoter wrote:
> hey mike,
>
> forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>
> -brian
>
> On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> at org.springframework.beans.factorysupport.AbstractBeanFactorygetBean(AbstractBeanFactory.java:241)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawlerjava:87)
>> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> at org.apache.oodt.cas.pge.PGETaskInstanceingestProducts(PGETaskInstance.java:349)
>> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStubjava:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>>
>> tasks.xml:
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I created
>>> >>> upon a successful ingestion of the output file(s). How do you configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://wwwspringframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> >>> ansxml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRessh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasagov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey chris,
this particular thing is a one liner... however, if there are current workflows using trunk CAS-PGE they may break due to the change... probably should talk more about maybe making this Java Property configurable or something
-brian
On Feb 08, 2012, at 06:54 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Thanks Bfost.
How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
I'll take a look at the end of this week, but won't have time before then, as I'm working
on Wengine porting, and some cas curator stuff that folks just reported.
Thanks!
Cheers,
Chris
On Feb 8, 2012, at 9:20 AM, holenoter wrote:
> hey mike,
>
> forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>
> -brian
>
> On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> at org.springframework.beans.factorysupport.AbstractBeanFactorygetBean(AbstractBeanFactory.java:241)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawlerjava:87)
>> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> at org.apache.oodt.cas.pge.PGETaskInstanceingestProducts(PGETaskInstance.java:349)
>> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStubjava:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>>
>> tasks.xml:
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I created
>>> >>> upon a successful ingestion of the output file(s). How do you configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://wwwspringframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> >>> ansxml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRessh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasagov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Bfost.
How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
I'll take a look at the end of this week, but won't have time before then, as I'm working
on Wengine porting, and some cas curator stuff that folks just reported.
Thanks!
Cheers,
Chris
On Feb 8, 2012, at 9:20 AM, holenoter wrote:
> hey mike,
>
> forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>
> -brian
>
> On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> at org.springframework.beans.factorysupport.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
>> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>>
>> tasks.xml:
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I created
>>> >>> upon a successful ingestion of the output file(s). How do you configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> >>> ansxml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Bfost.
How hard would it be to just port that part to trunk? Let's coordinate, me and you on it.
I'll take a look at the end of this week, but won't have time before then, as I'm working
on Wengine porting, and some cas curator stuff that folks just reported.
Thanks!
Cheers,
Chris
On Feb 8, 2012, at 9:20 AM, holenoter wrote:
> hey mike,
>
> forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
>
> -brian
>
> On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian and Chris,
>>
>> So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
>>
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
>> Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
>> WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
>> at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
>> at org.springframework.beans.factorysupport.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
>> at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
>> at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
>> at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
>> at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
>> at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
>> at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
>> at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
>> at orgapache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
>> at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
>> at java.lang.Thread.run(Thread.java:680)
>>
>>
>> I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
>>
>> <property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> and added a line in the PgeConfig.xml as Brian suggested:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
>>
>> and still got an error in the log as shown above.
>>
>> So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
>>
>> tasks.xml:
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
>>
>> PgeConfig.xml:
>>
>> <customMetadata>
>> <metadata key="PCS_ActionsIds" val="[Actions]"/>
>>
>>
>> Also, I found out that you can't have spaces in between the commas like so,
>>
>> <property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
>>
>>
>> Thanks for all your help guys!
>>
>>
>> -Mike
>>
>>
>> From: holenoter <ho...@mac.com>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 22:06:27 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Subject: Re: CAS PGE Crawler Question
>>
>> workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
>>
>> -brian
>>
>> On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> Space should work in the property value below. Can you show me
>>> the log output from CAS-PGE in the workflow log file (or the resource manager
>>> batch stubs?) It should show you what Actions are being executed
>>> and that can help to debug.
>>>
>>> Thanks!
>>>
>>> Cheers,
>>> Chris
>>>
>>> On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
>>>
>>> > If I wanted to specify more than one PCS_ActionsIds values, how would I do
>>> > this?
>>> >
>>> > I've tried the following with no success:
>>> >
>>> > <property name="PCS_ActionIds" value="Action1, Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1,Action2" />
>>> >
>>> >
>>> > and
>>> >
>>> > <property name="PCS_ActionIds" value="Action1 Action2" />
>>> >
>>> >
>>> >
>>> > -Mike
>>> >
>>> >
>>> > On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
>>> > <ch...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hi Mike,
>>> >>
>>> >> Glad to hear!
>>> >>
>>> >> Cheers,
>>> >> Chris
>>> >>
>>> >> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>> >>
>>> >>> Hi all,
>>> >>>
>>> >>> Disregard my previous message. Turns out I had a typo. I had to set
>>> >>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>> >>>
>>> >>> -Mike
>>> >>>
>>> >>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> >>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> >>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> >>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> >>> Subject: CAS PGE Crawler Question
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> I would like CAS PGE to run a post ingest success action that I created
>>> >>> upon a successful ingestion of the output file(s). How do you configure
>>> >>> CAS PGE for this type of scenario?
>>> >>>
>>> >>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> >>> action under the 'PCS_ActionIds' property:
>>> >>>
>>> >>> <task id="urn:my:Task" name="Task"
>>> >>> ...
>>> >>> <property name="PCS_ActionIds"
>>> >>> value="MyPostIngestSuccessAction" />
>>> >>> <property name="PCS_ActionRepoFile"
>>> >>> value="crawler-config.xml" />
>>> >>> </configuration>
>>> >>> </task>
>>> >>>
>>> >>> My crawler-config.xml has the following:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xmlns:p="http://www.springframework.org/schema/p"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> >>> <import resource="./action-beans.xml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> >>> ansxml"/>
>>> >>> <import
>>> >>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> >>> ns.xml"/>
>>> >>> </beans>
>>> >>>
>>> >>> Within my action-beans.xml, I added the post ingest success action:
>>> >>>
>>> >>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> >>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> >>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>> >>>
>>> >>> <bean
>>> >>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> >>> r"/>
>>> >>>
>>> >>> <!-- Crawler Actions -->
>>> >>>
>>> >>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> >>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> >>> <property name="description" value="Kickoff Another Workflow
>>> >>> Event"/>
>>> >>> <property name="executeCommand" value="sh
>>> >>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> >>> <property name="phases">
>>> >>> <list>
>>> >>> <value type="java.lang.String">postIngestSuccess</value>
>>> >>> </list>
>>> >>> </property>
>>> >>> </bean>
>>> >>>
>>> >>> Not sure if I'm missing anything else.
>>> >>>
>>> >>> Thanks in advance,
>>> >>> Mike
>>> >>>
>>> >>
>>> >>
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Chris Mattmann, Ph.D.
>>> >> Senior Computer Scientist
>>> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >> Office: 171-266B, Mailstop: 171-246
>>> >> Email: chris.a.mattmann@nasa.gov
>>> >> WWW: http://sunset.usc.edu/~mattmann/
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >> Adjunct Assistant Professor, Computer Science Department
>>> >> University of Southern California, Los Angeles, CA 90089 USA
>>> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>
>>> >
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey mike,
forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine precedence (or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
-brian
On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian and Chris,
So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
at org.apache.oodtcas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:680)
I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
<property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
and added a line in the PgeConfig.xml as Brian suggested:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
and still got an error in the log as shown above.
So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
tasks.xml:
<property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
PgeConfig.xml:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[Actions]"/>
Also, I found out that you can't have spaces in between the commas like so,
<property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
Thanks for all your help guys!
-Mike
From: holenoter <ho...@mac.com>
Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
Date: Tue, 7 Feb 2012 22:06:27 -0800
To: "user@oodt.apache.org" <us...@oodt.apache.org>
Subject: Re: CAS PGE Crawler Question
workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jplcrawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chrisa.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey mike,
forgot this didn't work with the trunk version... this was something that was changed in the branch... truck CAS-PGE uses PgeMetadata to determine�precedence�(or query order) when getting metadata values from static, local, or dynamic metadata... PgeMetadata's query order is: Static, Dynamic, then Custom (or Local)... the branched CAS-PGE uses WEngines ControlMetadata whose query order is: Dynamic, Local, then Static... so in the trunk there is no way to override static workflow metadata... something fixed in the branched version, and i guess chris can decide if this is something that he would like ported to the trunk in his porting efforts :)
-brian
On Feb 08, 2012, at 08:34 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian and Chris,
So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
at org.apache.oodtcas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:680)
I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
<property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
and added a line in the PgeConfig.xml as Brian suggested:
<customMetadata>
� <metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
and still got an error in the log as shown above.�
So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
tasks.xml:
<property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
PgeConfig.xml:
<customMetadata>
� <metadata key="PCS_ActionsIds" val="[Actions]"/>
Also, I found out that you can't have spaces in between the commas like so,
<property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
Thanks for all your help guys!�
-Mike
From: holenoter <ho...@mac.com>
Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
Date: Tue, 7 Feb 2012 22:06:27 -0800
To: "user@oodt.apache.org" <us...@oodt.apache.org>
Subject: Re: CAS PGE Crawler Question
workflow manager properties (static metadata) are passed to CAS-PGE via�WorkflowTaskConfiguration which only supports�scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jplcrawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chrisa.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian and Chris,
So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:680)
I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
<property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
and added a line in the PgeConfig.xml as Brian suggested:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
and still got an error in the log as shown above.
So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
tasks.xml:
<property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
PgeConfig.xml:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[Actions]"/>
Also, I found out that you can't have spaces in between the commas like so,
<property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
Thanks for all your help guys!
-Mike
From: holenoter <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tue, 7 Feb 2012 22:06:27 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: CAS PGE Crawler Question
workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov>> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>>
>>> Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd<http://www.springframework.org/schema/beans/spring-beans-25.xsd>">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian and Chris,
So, I first re-tried it using Chris' suggestion of using spaces and I still got an error. Here is what I'm seeing in the log:
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
INFO: Executing StdProductCrawler in productPath: [/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output]
Feb 8, 2012 7:55:33 AM org.apache.oodt.cas.pge.PGETaskInstance runIngestCrawler
WARNING: Failed while attempting to ingest products while crawling directory '/Users/mcayanan/smap/staging/L1A_RADAR_SPS/20120208T155531/output' (all products may not have been ingested) : No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 'KickoffL1BS0LoResWorkflow MoveMetadataToBackupDir' is defined
at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:360)
at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedBeanDefinition(AbstractBeanFactory.java:914)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:241)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:761)
at org.apache.oodt.cas.crawl.action.CrawlerActionRepo.loadActionsFromBeanFactory(CrawlerActionRepo.java:68)
at org.apache.oodt.cas.crawl.ProductCrawler.crawl(ProductCrawler.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.runIngestCrawler(PGETaskInstance.java:405)
at org.apache.oodt.cas.pge.PGETaskInstance.ingestProducts(PGETaskInstance.java:349)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:454)
at org.apache.oodt.cas.workflow.structs.TaskJob.execute(TaskJob.java:102)
at org.apache.oodt.cas.resource.system.extern.XmlRpcBatchStub$RunnableJob.run(XmlRpcBatchStub.java:229)
at java.lang.Thread.run(Thread.java:680)
I then modified my tasks.xml file where I specified the 'PCS_ActionsIds' property with the action IDs:
<property name="PCS_ActionsIds" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
and added a line in the PgeConfig.xml as Brian suggested:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[PCS_ActionsIds]"/>
and still got an error in the log as shown above.
So, I ended up re-naming the property name in the tasks.xml to some other name (so as to not reference itself essentially) and that did the trick!
tasks.xml:
<property name="Actions" value="KickoffL1BS0LoResWorkflow,MoveMetadataToBackupDir" />
PgeConfig.xml:
<customMetadata>
<metadata key="PCS_ActionsIds" val="[Actions]"/>
Also, I found out that you can't have spaces in between the commas like so,
<property name="Actions" value="KickoffL1BS0LoResWorkflow, MoveMetadataToBackupDir" />
Thanks for all your help guys!
-Mike
From: holenoter <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tue, 7 Feb 2012 22:06:27 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: CAS PGE Crawler Question
workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE.. so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov>> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>>
>>> Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd<http://www.springframework.org/schema/beans/spring-beans-25.xsd>">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
workflow manager properties (static metadata) are passed to CAS-PGE via�WorkflowTaskConfiguration which only supports�scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE... so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodtapache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ansxml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
workflow manager properties (static metadata) are passed to CAS-PGE via WorkflowTaskConfiguration which only supports scalar metadata... CAS-PGE doesn't re-parse out these metadata value because you have to allow for workflow metadata to contain values with commas without always being split out by CAS-PGE... so you have to explicitly repoint the metadata field at itself so it becomes custom (or local) metadata for CAS-PGE and then the value will be parsed on comma... context: CAS-PGE initially did the opposite (i.e. always re-parse workflow metadata into multi-values) but chris later changed this to the way it is now because this broke OCO since OCO had static workflow metadata which contained metadata with commas that was not supposed to be parsed into multi-values.
-brian
On Feb 07, 2012, at 07:36 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodtapache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ansxml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Space should work in the property value below. Can you show me
the log output from CAS-PGE in the workflow log file (or the resource manager
batch stubs?) It should show you what Actions are being executed
and that can help to debug.
Thanks!
Cheers,
Chris
On Feb 7, 2012, at 4:06 PM, Cayanan, Michael D (388J) wrote:
> If I wanted to specify more than one PCS_ActionsIds values, how would I do
> this?
>
> I've tried the following with no success:
>
> <property name="PCS_ActionIds" value="Action1, Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1,Action2" />
>
>
> and
>
> <property name="PCS_ActionIds" value="Action1 Action2" />
>
>
>
> -Mike
>
>
> On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> Glad to hear!
>>
>> Cheers,
>> Chris
>>
>> On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi all,
>>>
>>> Disregard my previous message. Turns out I had a typo. I had to set
>>> 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>>
>>> -Mike
>>>
>>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>>> Subject: CAS PGE Crawler Question
>>>
>>> Hi all,
>>>
>>> I would like CAS PGE to run a post ingest success action that I created
>>> upon a successful ingestion of the output file(s). How do you configure
>>> CAS PGE for this type of scenario?
>>>
>>> I have a tasks.xml file in the workflow, where I define my post ingest
>>> action under the 'PCS_ActionIds' property:
>>>
>>> <task id="urn:my:Task" name="Task"
>>> ...
>>> <property name="PCS_ActionIds"
>>> value="MyPostIngestSuccessAction" />
>>> <property name="PCS_ActionRepoFile"
>>> value="crawler-config.xml" />
>>> </configuration>
>>> </task>
>>>
>>> My crawler-config.xml has the following:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xmlns:p="http://www.springframework.org/schema/p"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>>> <import resource="./action-beans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>> ans.xml"/>
>>> <import
>>> resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>> ns.xml"/>
>>> </beans>
>>>
>>> Within my action-beans.xml, I added the post ingest success action:
>>>
>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>> http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>>
>>> <bean
>>> class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>> r"/>
>>>
>>> <!-- Crawler Actions -->
>>>
>>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>> class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>>> <property name="description" value="Kickoff Another Workflow
>>> Event"/>
>>> <property name="executeCommand" value="sh
>>> [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>>> <property name="phases">
>>> <list>
>>> <value type="java.lang.String">postIngestSuccess</value>
>>> </list>
>>> </property>
>>> </bean>
>>>
>>> Not sure if I'm missing anything else.
>>>
>>> Thanks in advance,
>>> Mike
>>>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey mike,
This one actually has to be set in you pge config xml file because workflow properties only support scalar values... you can leave it in your workflow config as you have it (using the commas) and just add this line to your pge config xml files:
<metadata key="PCS_ActionIds" val="[PCS_ActionIds]"/>
Basically as you have it right now it is treating the whole value attribute as a CrawlerAction ID.
-brian
On Feb 07, 2012, at 04:06 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
If I wanted to specify more than one PCS_ActionsIds values, how would I do
this?
I've tried the following with no success:
<property name="PCS_ActionIds" value="Action1, Action2" />
and
<property name="PCS_ActionIds" value="Action1,Action2" />
and
<property name="PCS_ActionIds" value="Action1 Action2" />
-Mike
On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>Glad to hear!
>
>Cheers,
>Chris
>
>On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi all,
>>
>> Disregard my previous message. Turns out I had a typo. I had to set
>>'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>
>> -Mike
>>
>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> Subject: CAS PGE Crawler Question
>>
>> Hi all,
>>
>> I would like CAS PGE to run a post ingest success action that I created
>>upon a successful ingestion of the output file(s). How do you configure
>>CAS PGE for this type of scenario?
>>
>> I have a tasks.xml file in the workflow, where I define my post ingest
>>action under the 'PCS_ActionIds' property:
>>
>> <task id="urn:my:Task" name="Task"
>> ...
>> <property name="PCS_ActionIds"
>>value="MyPostIngestSuccessAction" />
>> <property name="PCS_ActionRepoFile"
>>value="crawler-config.xml" />
>> </configuration>
>> </task>
>>
>> My crawler-config.xml has the following:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xmlns:p="http://www.springframework.org/schema/p"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> <import resource="./action-beans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>ans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>ns.xml"/>
>> </beans>
>>
>> Within my action-beans.xml, I added the post ingest success action:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <bean
>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>r"/>
>>
>> <!-- Crawler Actions -->
>>
>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> <property name="description" value="Kickoff Another Workflow
>>Event"/>
>> <property name="executeCommand" value="sh
>>[WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> <property name="phases">
>> <list>
>> <value type="java.lang.String">postIngestSuccess</value>
>> </list>
>> </property>
>> </bean>
>>
>> Not sure if I'm missing anything else.
>>
>> Thanks in advance,
>> Mike
>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by holenoter <ho...@mac.com>.
hey mike,
This one actually has to be set in you pge config xml file because workflow properties only support scalar values... you can leave it in your workflow config as you have it (using the commas) and just add this line to your pge config xml files:
<metadata key="PCS_ActionIds" val="[PCS_ActionIds]"/>
Basically as you have it right now it is treating the whole value attribute as a CrawlerAction ID.
-brian
On Feb 07, 2012, at 04:06 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
If I wanted to specify more than one PCS_ActionsIds values, how would I do
this?
I've tried the following with no success:
<property name="PCS_ActionIds" value="Action1, Action2" />
and
<property name="PCS_ActionIds" value="Action1,Action2" />
and
<property name="PCS_ActionIds" value="Action1 Action2" />
-Mike
On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>Glad to hear!
>
>Cheers,
>Chris
>
>On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi all,
>>
>> Disregard my previous message. Turns out I had a typo. I had to set
>>'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>
>> -Mike
>>
>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> Subject: CAS PGE Crawler Question
>>
>> Hi all,
>>
>> I would like CAS PGE to run a post ingest success action that I created
>>upon a successful ingestion of the output file(s). How do you configure
>>CAS PGE for this type of scenario?
>>
>> I have a tasks.xml file in the workflow, where I define my post ingest
>>action under the 'PCS_ActionIds' property:
>>
>> <task id="urn:my:Task" name="Task"
>> ...
>> <property name="PCS_ActionIds"
>>value="MyPostIngestSuccessAction" />
>> <property name="PCS_ActionRepoFile"
>>value="crawler-config.xml" />
>> </configuration>
>> </task>
>>
>> My crawler-config.xml has the following:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xmlns:p="http://www.springframework.org/schema/p"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> <import resource="./action-beans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>ans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>ns.xml"/>
>> </beans>
>>
>> Within my action-beans.xml, I added the post ingest success action:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <bean
>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>r"/>
>>
>> <!-- Crawler Actions -->
>>
>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> <property name="description" value="Kickoff Another Workflow
>>Event"/>
>> <property name="executeCommand" value="sh
>>[WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> <property name="phases">
>> <list>
>> <value type="java.lang.String">postIngestSuccess</value>
>> </list>
>> </property>
>> </bean>
>>
>> Not sure if I'm missing anything else.
>>
>> Thanks in advance,
>> Mike
>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
If I wanted to specify more than one PCS_ActionsIds values, how would I do
this?
I've tried the following with no success:
<property name="PCS_ActionIds" value="Action1, Action2" />
and
<property name="PCS_ActionIds" value="Action1,Action2" />
and
<property name="PCS_ActionIds" value="Action1 Action2" />
-Mike
On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>Glad to hear!
>
>Cheers,
>Chris
>
>On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi all,
>>
>> Disregard my previous message. Turns out I had a typo. I had to set
>>'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>
>> -Mike
>>
>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> Subject: CAS PGE Crawler Question
>>
>> Hi all,
>>
>> I would like CAS PGE to run a post ingest success action that I created
>>upon a successful ingestion of the output file(s). How do you configure
>>CAS PGE for this type of scenario?
>>
>> I have a tasks.xml file in the workflow, where I define my post ingest
>>action under the 'PCS_ActionIds' property:
>>
>> <task id="urn:my:Task" name="Task"
>> ...
>> <property name="PCS_ActionIds"
>>value="MyPostIngestSuccessAction" />
>> <property name="PCS_ActionRepoFile"
>>value="crawler-config.xml" />
>> </configuration>
>> </task>
>>
>> My crawler-config.xml has the following:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xmlns:p="http://www.springframework.org/schema/p"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> <import resource="./action-beans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>ans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>ns.xml"/>
>> </beans>
>>
>> Within my action-beans.xml, I added the post ingest success action:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <bean
>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>r"/>
>>
>> <!-- Crawler Actions -->
>>
>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> <property name="description" value="Kickoff Another Workflow
>>Event"/>
>> <property name="executeCommand" value="sh
>>[WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> <property name="phases">
>> <list>
>> <value type="java.lang.String">postIngestSuccess</value>
>> </list>
>> </property>
>> </bean>
>>
>> Not sure if I'm missing anything else.
>>
>> Thanks in advance,
>> Mike
>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
If I wanted to specify more than one PCS_ActionsIds values, how would I do
this?
I've tried the following with no success:
<property name="PCS_ActionIds" value="Action1, Action2" />
and
<property name="PCS_ActionIds" value="Action1,Action2" />
and
<property name="PCS_ActionIds" value="Action1 Action2" />
-Mike
On 2/7/12 3:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>Glad to hear!
>
>Cheers,
>Chris
>
>On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi all,
>>
>> Disregard my previous message. Turns out I had a typo. I had to set
>>'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>>
>> -Mike
>>
>> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
>> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Date: Tue, 7 Feb 2012 13:51:12 -0800
>> To: "user@oodt.apache.org" <us...@oodt.apache.org>
>> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
>> Subject: CAS PGE Crawler Question
>>
>> Hi all,
>>
>> I would like CAS PGE to run a post ingest success action that I created
>>upon a successful ingestion of the output file(s). How do you configure
>>CAS PGE for this type of scenario?
>>
>> I have a tasks.xml file in the workflow, where I define my post ingest
>>action under the 'PCS_ActionIds' property:
>>
>> <task id="urn:my:Task" name="Task"
>> ...
>> <property name="PCS_ActionIds"
>>value="MyPostIngestSuccessAction" />
>> <property name="PCS_ActionRepoFile"
>>value="crawler-config.xml" />
>> </configuration>
>> </task>
>>
>> My crawler-config.xml has the following:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xmlns:p="http://www.springframework.org/schema/p"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
>> <import resource="./action-beans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-be
>>ans.xml"/>
>> <import
>>resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-bea
>>ns.xml"/>
>> </beans>
>>
>> Within my action-beans.xml, I added the post ingest success action:
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans
>>http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>>
>> <bean
>>class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcesso
>>r"/>
>>
>> <!-- Crawler Actions -->
>>
>> <bean id="MyPostIngestSuccessAction" lazy-init="true"
>>class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
>> <property name="description" value="Kickoff Another Workflow
>>Event"/>
>> <property name="executeCommand" value="sh
>>[WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
>> <property name="phases">
>> <list>
>> <value type="java.lang.String">postIngestSuccess</value>
>> </list>
>> </property>
>> </bean>
>>
>> Not sure if I'm missing anything else.
>>
>> Thanks in advance,
>> Mike
>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Glad to hear!
Cheers,
Chris
On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
> Hi all,
>
> Disregard my previous message. Turns out I had a typo. I had to set 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>
> -Mike
>
> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Tue, 7 Feb 2012 13:51:12 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
> Subject: CAS PGE Crawler Question
>
> Hi all,
>
> I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
>
> I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
>
> <task id="urn:my:Task" name="Task"
> ...
> <property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
> <property name="PCS_ActionRepoFile" value="crawler-config.xml" />
> </configuration>
> </task>
>
> My crawler-config.xml has the following:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:p="http://www.springframework.org/schema/p"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
> <import resource="./action-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
> </beans>
>
> Within my action-beans.xml, I added the post ingest success action:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
>
> <!-- Crawler Actions -->
>
> <bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
> <property name="description" value="Kickoff Another Workflow Event"/>
> <property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
> <property name="phases">
> <list>
> <value type="java.lang.String">postIngestSuccess</value>
> </list>
> </property>
> </bean>
>
> Not sure if I'm missing anything else.
>
> Thanks in advance,
> Mike
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Glad to hear!
Cheers,
Chris
On Feb 7, 2012, at 3:14 PM, Cayanan, Michael D (388J) wrote:
> Hi all,
>
> Disregard my previous message. Turns out I had a typo. I had to set 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
>
> -Mike
>
> From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Tue, 7 Feb 2012 13:51:12 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>
> Subject: CAS PGE Crawler Question
>
> Hi all,
>
> I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
>
> I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
>
> <task id="urn:my:Task" name="Task"
> ...
> <property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
> <property name="PCS_ActionRepoFile" value="crawler-config.xml" />
> </configuration>
> </task>
>
> My crawler-config.xml has the following:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:p="http://www.springframework.org/schema/p"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
> <import resource="./action-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
> <import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
> </beans>
>
> Within my action-beans.xml, I added the post ingest success action:
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd">
>
> <bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
>
> <!-- Crawler Actions -->
>
> <bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
> <property name="description" value="Kickoff Another Workflow Event"/>
> <property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
> <property name="phases">
> <list>
> <value type="java.lang.String">postIngestSuccess</value>
> </list>
> </property>
> </bean>
>
> Not sure if I'm missing anything else.
>
> Thanks in advance,
> Mike
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi all,
Disregard my previous message. Turns out I had a typo. I had to set 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
-Mike
From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tue, 7 Feb 2012 13:51:12 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>>
Subject: CAS PGE Crawler Question
Hi all,
I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
<task id="urn:my:Task" name="Task"
...
<property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
<property name="PCS_ActionRepoFile" value="crawler-config.xml" />
</configuration>
</task>
My crawler-config.xml has the following:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
<import resource="./action-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
</beans>
Within my action-beans.xml, I added the post ingest success action:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
<!-- Crawler Actions -->
<bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
<property name="description" value="Kickoff Another Workflow Event"/>
<property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
<property name="phases">
<list>
<value type="java.lang.String">postIngestSuccess</value>
</list>
</property>
</bean>
Not sure if I'm missing anything else.
Thanks in advance,
Mike
Re: CAS PGE Crawler Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi all,
Disregard my previous message. Turns out I had a typo. I had to set 'PCS_ActionsIds' instead of 'PCS_ActionIds'.
-Mike
From: "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tue, 7 Feb 2012 13:51:12 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Cc: "Wong, Cynthia L (388J)" <cy...@jpl.nasa.gov>>
Subject: CAS PGE Crawler Question
Hi all,
I would like CAS PGE to run a post ingest success action that I created upon a successful ingestion of the output file(s). How do you configure CAS PGE for this type of scenario?
I have a tasks.xml file in the workflow, where I define my post ingest action under the 'PCS_ActionIds' property:
<task id="urn:my:Task" name="Task"
...
<property name="PCS_ActionIds" value="MyPostIngestSuccessAction" />
<property name="PCS_ActionRepoFile" value="crawler-config.xml" />
</configuration>
</task>
My crawler-config.xml has the following:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<import resource="classpath:/org/apache/oodt/cas/crawl/crawler-beans.xml"/>
<import resource="./action-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/comparator/precondition-beans.xml"/>
<import resource="classpath:/org/apache/oodt/cas/crawl/option/cmd-line-option-beans.xml"/>
</beans>
Within my action-beans.xml, I added the post ingest success action:
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.5.xsd"<http://www.springframework.org/schema/beans/spring-beans-2.5.xsd%22>>
<bean class="org.apache.oodt.commons.spring.postprocessor.SetIdBeanPostProcessor"/>
<!-- Crawler Actions -->
<bean id="MyPostIngestSuccessAction" lazy-init="true" class="gov.nasa.jpl.crawler.action.MyPostIngestSuccessAction">
<property name="description" value="Kickoff Another Workflow Event"/>
<property name="executeCommand" value="sh [WORKFLOWMGR_HOME]/bin/run_L1B_S0_LoRes.sh"/>
<property name="phases">
<list>
<value type="java.lang.String">postIngestSuccess</value>
</list>
</property>
</bean>
Not sure if I'm missing anything else.
Thanks in advance,
Mike