You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oodt.apache.org by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> on 2012/03/02 21:36:04 UTC
PGE Question
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Anytime, man!
Cheers,
Chris
On Mar 3, 2012, at 1:18 AM, Cayanan, Michael D (388J) wrote:
> Thanks Chris. Actually after I sent this message out, I realized that the
> easier way is to just store the workflow instance id in the file manager
> and querying that for the ingested files (see Brian's reply).
>
> In any case, this is good to know should I ever need to do this in the
> future!
>
> Cheers,
> Mike
>
> On 3/2/12 10:21 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi,
>>>
>>> In my PGE, I specify my output like the following:
>>>
>>> <output>
>>> <dir path="[ProductLocation]" createBeforeExe="true">
>>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5"
>>> metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>>> <files
>>> name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"
>>> metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>>> </dir>
>>> </output>
>>>
>>> After ingestion of this output product, I'd like to save the filename
>>> in my workflow metadata. How would I do this?
>>
>> You could extend CAS-PGE, and override the protected void
>> runIngestCrawler(StdProductCrawler crawler,
>> List<File> crawlDirs) method like so:
>>
>> public class SMAPPge extends StdPGETaskInstance{
>>
>> @Override
>> protected void runIngestCrawler(StdProductCrawler crawler,
>> List<File> crawlDirs) {
>> super.runIngestCrawler(crawler, crawlDirs);
>>
>> for(IngestStatus status: crawler.getIngestStatus()){
>> this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status.
>> getProduct().getName());
>> }
>>
>> }
>>
>> See this file: http://s.apache.org/ym
>>
>>>
>>> I tried creating a PostIngestSuccess action, but found out that
>>> whatever metadata saving you want to do on the "crawler" side doesn't
>>> get passed back to the workflow metadata.
>>
>> That's by design. Crawler actions are meant to perform functionality
>> "using" a read-only copy of the workflow met. They are *not* intended to
>> modify workflow met.
>>
>>>
>>> I suppose I could update the workflow metadata by creating a
>>> PostIngestSuccess action and using the workflow manager client API to do
>>> this. However, when I look at this class, I see the following method,
>>>
>>>
>>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>>
>>>
>>> where it requires the workflow instance ID as a parameter. Is there a
>>> way to capture the workflow instance ID into the workflow metadata so
>>> that I can leverage this method to update my metadata?
>>
>> You've got the workflow instance id. It's given to you by the workflow
>> manager in the metadata context for each workflow instance, under the key
>> name:
>>
>> "WorkflowInstId"
>>
>> See here: http://s.apache.org/NNs
>>
>> and search for "getSharedContext()" for the list of keys that are
>> automatically set.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Anytime, man!
Cheers,
Chris
On Mar 3, 2012, at 1:18 AM, Cayanan, Michael D (388J) wrote:
> Thanks Chris. Actually after I sent this message out, I realized that the
> easier way is to just store the workflow instance id in the file manager
> and querying that for the ingested files (see Brian's reply).
>
> In any case, this is good to know should I ever need to do this in the
> future!
>
> Cheers,
> Mike
>
> On 3/2/12 10:21 PM, "Mattmann, Chris A (388J)"
> <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi,
>>>
>>> In my PGE, I specify my output like the following:
>>>
>>> <output>
>>> <dir path="[ProductLocation]" createBeforeExe="true">
>>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5"
>>> metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>>> <files
>>> name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"
>>> metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>>> </dir>
>>> </output>
>>>
>>> After ingestion of this output product, I'd like to save the filename
>>> in my workflow metadata. How would I do this?
>>
>> You could extend CAS-PGE, and override the protected void
>> runIngestCrawler(StdProductCrawler crawler,
>> List<File> crawlDirs) method like so:
>>
>> public class SMAPPge extends StdPGETaskInstance{
>>
>> @Override
>> protected void runIngestCrawler(StdProductCrawler crawler,
>> List<File> crawlDirs) {
>> super.runIngestCrawler(crawler, crawlDirs);
>>
>> for(IngestStatus status: crawler.getIngestStatus()){
>> this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status.
>> getProduct().getName());
>> }
>>
>> }
>>
>> See this file: http://s.apache.org/ym
>>
>>>
>>> I tried creating a PostIngestSuccess action, but found out that
>>> whatever metadata saving you want to do on the "crawler" side doesn't
>>> get passed back to the workflow metadata.
>>
>> That's by design. Crawler actions are meant to perform functionality
>> "using" a read-only copy of the workflow met. They are *not* intended to
>> modify workflow met.
>>
>>>
>>> I suppose I could update the workflow metadata by creating a
>>> PostIngestSuccess action and using the workflow manager client API to do
>>> this. However, when I look at this class, I see the following method,
>>>
>>>
>>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>>
>>>
>>> where it requires the workflow instance ID as a parameter. Is there a
>>> way to capture the workflow instance ID into the workflow metadata so
>>> that I can leverage this method to update my metadata?
>>
>> You've got the workflow instance id. It's given to you by the workflow
>> manager in the metadata context for each workflow instance, under the key
>> name:
>>
>> "WorkflowInstId"
>>
>> See here: http://s.apache.org/NNs
>>
>> and search for "getSharedContext()" for the list of keys that are
>> automatically set.
>>
>> Cheers,
>> Chris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Thanks Chris. Actually after I sent this message out, I realized that the
easier way is to just store the workflow instance id in the file manager
and querying that for the ingested files (see Brian's reply).
In any case, this is good to know should I ever need to do this in the
future!
Cheers,
Mike
On 3/2/12 10:21 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5"
>>metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files
>>name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"
>>metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename
>>in my workflow metadata. How would I do this?
>
>You could extend CAS-PGE, and override the protected void
>runIngestCrawler(StdProductCrawler crawler,
> List<File> crawlDirs) method like so:
>
>public class SMAPPge extends StdPGETaskInstance{
>
> @Override
>protected void runIngestCrawler(StdProductCrawler crawler,
> List<File> crawlDirs) {
> super.runIngestCrawler(crawler, crawlDirs);
>
> for(IngestStatus status: crawler.getIngestStatus()){
> this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status.
>getProduct().getName());
> }
>
> }
>
>See this file: http://s.apache.org/ym
>
>>
>> I tried creating a PostIngestSuccess action, but found out that
>>whatever metadata saving you want to do on the "crawler" side doesn't
>>get passed back to the workflow metadata.
>
>That's by design. Crawler actions are meant to perform functionality
>"using" a read-only copy of the workflow met. They are *not* intended to
>modify workflow met.
>
>>
>> I suppose I could update the workflow metadata by creating a
>>PostIngestSuccess action and using the workflow manager client API to do
>>this. However, when I look at this class, I see the following method,
>>
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a
>>way to capture the workflow instance ID into the workflow metadata so
>>that I can leverage this method to update my metadata?
>
>You've got the workflow instance id. It's given to you by the workflow
>manager in the metadata context for each workflow instance, under the key
>name:
>
>"WorkflowInstId"
>
>See here: http://s.apache.org/NNs
>
>and search for "getSharedContext()" for the list of keys that are
>automatically set.
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Thanks Chris. Actually after I sent this message out, I realized that the
easier way is to just store the workflow instance id in the file manager
and querying that for the ingested files (see Brian's reply).
In any case, this is good to know should I ever need to do this in the
future!
Cheers,
Mike
On 3/2/12 10:21 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5"
>>metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files
>>name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"
>>metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename
>>in my workflow metadata. How would I do this?
>
>You could extend CAS-PGE, and override the protected void
>runIngestCrawler(StdProductCrawler crawler,
> List<File> crawlDirs) method like so:
>
>public class SMAPPge extends StdPGETaskInstance{
>
> @Override
>protected void runIngestCrawler(StdProductCrawler crawler,
> List<File> crawlDirs) {
> super.runIngestCrawler(crawler, crawlDirs);
>
> for(IngestStatus status: crawler.getIngestStatus()){
> this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status.
>getProduct().getName());
> }
>
> }
>
>See this file: http://s.apache.org/ym
>
>>
>> I tried creating a PostIngestSuccess action, but found out that
>>whatever metadata saving you want to do on the "crawler" side doesn't
>>get passed back to the workflow metadata.
>
>That's by design. Crawler actions are meant to perform functionality
>"using" a read-only copy of the workflow met. They are *not* intended to
>modify workflow met.
>
>>
>> I suppose I could update the workflow metadata by creating a
>>PostIngestSuccess action and using the workflow manager client API to do
>>this. However, when I look at this class, I see the following method,
>>
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a
>>way to capture the workflow instance ID into the workflow metadata so
>>that I can leverage this method to update my metadata?
>
>You've got the workflow instance id. It's given to you by the workflow
>manager in the metadata context for each workflow instance, under the key
>name:
>
>"WorkflowInstId"
>
>See here: http://s.apache.org/NNs
>
>and search for "getSharedContext()" for the list of keys that are
>automatically set.
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
> Hi,
>
> In my PGE, I specify my output like the following:
>
> <output>
> <dir path="[ProductLocation]" createBeforeExe="true">
> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
> </dir>
> </output>
>
> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
You could extend CAS-PGE, and override the protected void runIngestCrawler(StdProductCrawler crawler,
List<File> crawlDirs) method like so:
public class SMAPPge extends StdPGETaskInstance{
@Override
protected void runIngestCrawler(StdProductCrawler crawler,
List<File> crawlDirs) {
super.runIngestCrawler(crawler, crawlDirs);
for(IngestStatus status: crawler.getIngestStatus()){
this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status. getProduct().getName());
}
}
See this file: http://s.apache.org/ym
>
> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
That's by design. Crawler actions are meant to perform functionality "using" a read-only copy of the workflow met. They are *not* intended to
modify workflow met.
>
> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>
> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>
>
> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
You've got the workflow instance id. It's given to you by the workflow manager in the metadata context for each workflow instance, under the key name:
"WorkflowInstId"
See here: http://s.apache.org/NNs
and search for "getSharedContext()" for the list of keys that are automatically set.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by Sheryl John <sh...@gmail.com>.
Hi Mike,
Sorry for the delay. I had some problems with my machine when I was
typing my reply earlier.
Yes, we've done the same thing that Chris said above to add your new
key + val to the existing workflow met.
--
-Sheryl
Re: PGE Question
Posted by Sheryl John <sh...@gmail.com>.
Hi Mike,
Sorry for the delay. I had some problems with my machine when I was
typing my reply earlier.
Yes, we've done the same thing that Chris said above to add your new
key + val to the existing workflow met.
--
-Sheryl
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On Mar 2, 2012, at 3:43 PM, Cayanan, Michael D (388J) wrote:
> Hi Sheryl,
>
> So, I took your suggestion and put this piece of code together in a post success ingest crawler action:
>
> String key = "L0b_Radar_Product_Filename";
> LOG.log(Level.INFO, "Workflow metadata keys: " + metadata.getAllKeys());
>
> Metadata productNameMet = new Metadata();
> productNameMet.addMetadata(key, product.getName());
> try {
> XmlRpcWorkflowManagerClient wClient = new XmlRpcWorkflowManagerClient(
> new URL(metadata.getMetadata(CoreMetKeys.WORKFLOW_MANAGER_URL)));
> passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
> if (passFlag) {
> LOG.log(Level.INFO, "Updated workflow id '" + metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID) + "' with metadata: " + productNameMet.getAllKeys());
> Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
> LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
>
>
>
> I was hoping that this would update my workflow metadata, however it doesn't appear to be working. I'm seeing this in my logs:
>
> INFO: Workflow metadata keys: [FileLocation, TemporalRange~RangeEndingTime, ProductPathGroup~FileVersionId, StaticAncillaryFileGroup~L0b_Radar_MetDefFile, TemporalRange~RangeBeginningDate, WorkflowManagerUrl, TaskId, Filename, InputFileGroup~InputFilePath, StaticAncillaryFileGroup~L0b_Radar_ProductConfig, TemporalRange~RangeBeginningTime, WorkflowInstId, TemporalRange~RangeEndingDate, ProductType, JobId, ProductName, StaticAncillaryFileGroup~L0b_Radar_InputConfig, L0a_Radar_Product_Filename, ProcessingNode]
> Mar 2, 2012 3:28:21 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
> INFO: Updated workflow id '6333dd86-64bf-11e1-98c2-afe386bb7e05' with metadata: [L0b_Radar_Product_Filename]
> Mar 2, 2012 3:28:22 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
> INFO: Metadata after update: [L0b_Radar_Product_Filename]
>
> Note that even getting the metadata via the workflow client API produces an odd result.
Not sure why this is odd.
In line: String key = "L0b_Radar_Product_Filename";
You set the key name to: L0b_Radar_Product_Filename
Then, in line: Metadata productNameMet = new Metadata();
You create a new met object.
Then in line: productNameMet.addMetadata(key, product.getName());
You add a single key to that new met object with the product value.
Then in line: passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
You update the workflow instance with the ID, passing it the productNameMet, thus replacing the metadata for this workflow instance, with
the one with only a single key.
Then in line: Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
You get the metadata for the workflow which should now only have 1 key.
And then in line: LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
You print the key names and you see 1 key name.
> It's only returning the single metadata key that I wanted to add to the current metadata in the workflow.
To do that you need to merge your new product met key + val with the existing workflow met. Try doing a get Workflow Instance metadata first, then
appending your key to it, then saving that back.
> Not sure why that is. In any case, when I look at the metadata in the task that follows this PGE, I only see the metadata without the extra metadata key that I wished to add.
>
> It would seem like I'm doing this correctly, however I'm unsure if I'm missing something or if this is a bug in the code.
See above, Mike. It's operating as expected but you just need to update it to plumb through the original met too.
HTH!
Cheers,
Chris
>
> -Mike
>
> From: Sheryl John <sh...@gmail.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Fri, 2 Mar 2012 12:58:53 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: PGE Question
>
> Hi Mike,
>
> You can use any of the Workflow Core met keys inside your PGE.
> So, if you want to use the WorkflowInstId, you would specify it like this: [WorkflowInstId] or [JobId] inside your PGE script.
>
> But there could be other ways to update workflow metadata. You could also have this in you metFileWriterClass, but that will be before your file is ingested.
>
> On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <mi...@jpl.nasa.gov> wrote:
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
>>
>> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
>>
>> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
>>
>> Thanks,
>> Mike
>>
>>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On Mar 2, 2012, at 3:43 PM, Cayanan, Michael D (388J) wrote:
> Hi Sheryl,
>
> So, I took your suggestion and put this piece of code together in a post success ingest crawler action:
>
> String key = "L0b_Radar_Product_Filename";
> LOG.log(Level.INFO, "Workflow metadata keys: " + metadata.getAllKeys());
>
> Metadata productNameMet = new Metadata();
> productNameMet.addMetadata(key, product.getName());
> try {
> XmlRpcWorkflowManagerClient wClient = new XmlRpcWorkflowManagerClient(
> new URL(metadata.getMetadata(CoreMetKeys.WORKFLOW_MANAGER_URL)));
> passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
> if (passFlag) {
> LOG.log(Level.INFO, "Updated workflow id '" + metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID) + "' with metadata: " + productNameMet.getAllKeys());
> Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
> LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
>
>
>
> I was hoping that this would update my workflow metadata, however it doesn't appear to be working. I'm seeing this in my logs:
>
> INFO: Workflow metadata keys: [FileLocation, TemporalRange~RangeEndingTime, ProductPathGroup~FileVersionId, StaticAncillaryFileGroup~L0b_Radar_MetDefFile, TemporalRange~RangeBeginningDate, WorkflowManagerUrl, TaskId, Filename, InputFileGroup~InputFilePath, StaticAncillaryFileGroup~L0b_Radar_ProductConfig, TemporalRange~RangeBeginningTime, WorkflowInstId, TemporalRange~RangeEndingDate, ProductType, JobId, ProductName, StaticAncillaryFileGroup~L0b_Radar_InputConfig, L0a_Radar_Product_Filename, ProcessingNode]
> Mar 2, 2012 3:28:21 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
> INFO: Updated workflow id '6333dd86-64bf-11e1-98c2-afe386bb7e05' with metadata: [L0b_Radar_Product_Filename]
> Mar 2, 2012 3:28:22 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
> INFO: Metadata after update: [L0b_Radar_Product_Filename]
>
> Note that even getting the metadata via the workflow client API produces an odd result.
Not sure why this is odd.
In line: String key = "L0b_Radar_Product_Filename";
You set the key name to: L0b_Radar_Product_Filename
Then, in line: Metadata productNameMet = new Metadata();
You create a new met object.
Then in line: productNameMet.addMetadata(key, product.getName());
You add a single key to that new met object with the product value.
Then in line: passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
You update the workflow instance with the ID, passing it the productNameMet, thus replacing the metadata for this workflow instance, with
the one with only a single key.
Then in line: Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
You get the metadata for the workflow which should now only have 1 key.
And then in line: LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
You print the key names and you see 1 key name.
> It's only returning the single metadata key that I wanted to add to the current metadata in the workflow.
To do that you need to merge your new product met key + val with the existing workflow met. Try doing a get Workflow Instance metadata first, then
appending your key to it, then saving that back.
> Not sure why that is. In any case, when I look at the metadata in the task that follows this PGE, I only see the metadata without the extra metadata key that I wished to add.
>
> It would seem like I'm doing this correctly, however I'm unsure if I'm missing something or if this is a bug in the code.
See above, Mike. It's operating as expected but you just need to update it to plumb through the original met too.
HTH!
Cheers,
Chris
>
> -Mike
>
> From: Sheryl John <sh...@gmail.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Fri, 2 Mar 2012 12:58:53 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: PGE Question
>
> Hi Mike,
>
> You can use any of the Workflow Core met keys inside your PGE.
> So, if you want to use the WorkflowInstId, you would specify it like this: [WorkflowInstId] or [JobId] inside your PGE script.
>
> But there could be other ways to update workflow metadata. You could also have this in you metFileWriterClass, but that will be before your file is ingested.
>
> On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <mi...@jpl.nasa.gov> wrote:
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
>>
>> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
>>
>> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
>>
>> Thanks,
>> Mike
>>
>>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Sheryl,
So, I took your suggestion and put this piece of code together in a post success ingest crawler action:
String key = "L0b_Radar_Product_Filename";
LOG.log(Level.INFO, "Workflow metadata keys: " + metadata.getAllKeys());
Metadata productNameMet = new Metadata();
productNameMet.addMetadata(key, product.getName());
try {
XmlRpcWorkflowManagerClient wClient = new XmlRpcWorkflowManagerClient(
new URL(metadata.getMetadata(CoreMetKeys.WORKFLOW_MANAGER_URL)));
passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
if (passFlag) {
LOG.log(Level.INFO, "Updated workflow id '" + metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID) + "' with metadata: " + productNameMet.getAllKeys());
Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
I was hoping that this would update my workflow metadata, however it doesn't appear to be working. I'm seeing this in my logs:
INFO: Workflow metadata keys: [FileLocation, TemporalRange~RangeEndingTime, ProductPathGroup~FileVersionId, StaticAncillaryFileGroup~L0b_Radar_MetDefFile, TemporalRange~RangeBeginningDate, WorkflowManagerUrl, TaskId, Filename, InputFileGroup~InputFilePath, StaticAncillaryFileGroup~L0b_Radar_ProductConfig, TemporalRange~RangeBeginningTime, WorkflowInstId, TemporalRange~RangeEndingDate, ProductType, JobId, ProductName, StaticAncillaryFileGroup~L0b_Radar_InputConfig, L0a_Radar_Product_Filename, ProcessingNode]
Mar 2, 2012 3:28:21 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
INFO: Updated workflow id '6333dd86-64bf-11e1-98c2-afe386bb7e05' with metadata: [L0b_Radar_Product_Filename]
Mar 2, 2012 3:28:22 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
INFO: Metadata after update: [L0b_Radar_Product_Filename]
Note that even getting the metadata via the workflow client API produces an odd result. It's only returning the single metadata key that I wanted to add to the current metadata in the workflow. Not sure why that is. In any case, when I look at the metadata in the task that follows this PGE, I only see the metadata without the extra metadata key that I wished to add.
It would seem like I'm doing this correctly, however I'm unsure if I'm missing something or if this is a bug in the code.
-Mike
From: Sheryl John <sh...@gmail.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Fri, 2 Mar 2012 12:58:53 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: PGE Question
Hi Mike,
You can use any of the Workflow Core met keys inside your PGE.
So, if you want to use the WorkflowInstId, you would specify it like this: [WorkflowInstId] or [JobId] inside your PGE script.
But there could be other ways to update workflow metadata. You could also have this in you metFileWriterClass, but that will be before your file is ingested.
On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <mi...@jpl.nasa.gov>> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
--
-Sheryl
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Sheryl,
So, I took your suggestion and put this piece of code together in a post success ingest crawler action:
String key = "L0b_Radar_Product_Filename";
LOG.log(Level.INFO, "Workflow metadata keys: " + metadata.getAllKeys());
Metadata productNameMet = new Metadata();
productNameMet.addMetadata(key, product.getName());
try {
XmlRpcWorkflowManagerClient wClient = new XmlRpcWorkflowManagerClient(
new URL(metadata.getMetadata(CoreMetKeys.WORKFLOW_MANAGER_URL)));
passFlag = wClient.updateMetadataForWorkflow(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID), productNameMet);
if (passFlag) {
LOG.log(Level.INFO, "Updated workflow id '" + metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID) + "' with metadata: " + productNameMet.getAllKeys());
Metadata after = wClient.getWorkflowInstanceMetadata(metadata.getMetadata(CoreMetKeys.WORKFLOW_INST_ID));
LOG.log(Level.INFO, "Metadata after update: " + after.getAllKeys());
I was hoping that this would update my workflow metadata, however it doesn't appear to be working. I'm seeing this in my logs:
INFO: Workflow metadata keys: [FileLocation, TemporalRange~RangeEndingTime, ProductPathGroup~FileVersionId, StaticAncillaryFileGroup~L0b_Radar_MetDefFile, TemporalRange~RangeBeginningDate, WorkflowManagerUrl, TaskId, Filename, InputFileGroup~InputFilePath, StaticAncillaryFileGroup~L0b_Radar_ProductConfig, TemporalRange~RangeBeginningTime, WorkflowInstId, TemporalRange~RangeEndingDate, ProductType, JobId, ProductName, StaticAncillaryFileGroup~L0b_Radar_InputConfig, L0a_Radar_Product_Filename, ProcessingNode]
Mar 2, 2012 3:28:21 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
INFO: Updated workflow id '6333dd86-64bf-11e1-98c2-afe386bb7e05' with metadata: [L0b_Radar_Product_Filename]
Mar 2, 2012 3:28:22 PM gov.nasa.smap.spdm.crawl.action.AddProductNameToWorkflowMetAction performAction
INFO: Metadata after update: [L0b_Radar_Product_Filename]
Note that even getting the metadata via the workflow client API produces an odd result. It's only returning the single metadata key that I wanted to add to the current metadata in the workflow. Not sure why that is. In any case, when I look at the metadata in the task that follows this PGE, I only see the metadata without the extra metadata key that I wished to add.
It would seem like I'm doing this correctly, however I'm unsure if I'm missing something or if this is a bug in the code.
-Mike
From: Sheryl John <sh...@gmail.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Fri, 2 Mar 2012 12:58:53 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: PGE Question
Hi Mike,
You can use any of the Workflow Core met keys inside your PGE.
So, if you want to use the WorkflowInstId, you would specify it like this: [WorkflowInstId] or [JobId] inside your PGE script.
But there could be other ways to update workflow metadata. You could also have this in you metFileWriterClass, but that will be before your file is ingested.
On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <mi...@jpl.nasa.gov>> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
--
-Sheryl
Re: PGE Question
Posted by Sheryl John <sh...@gmail.com>.
Hi Mike,
You can use any of the Workflow Core met keys inside your PGE.
So, if you want to use the WorkflowInstId, you would specify it like this:
[WorkflowInstId] or [JobId] inside your PGE script.
But there could be other ways to update workflow metadata. You could also
have this in you metFileWriterClass, but that will be before your file is
ingested.
On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <
michael.d.cayanan@jpl.nasa.gov> wrote:
> Hi,
>
> In my PGE, I specify my output like the following:
>
> <output>
>
> <dir path="[ProductLocation]" createBeforeExe="true">
>
> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass=
> "[MetFileWriter]" args="[ConfigMetoutFile]" />
>
> <files name=
> "[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass=
> "[MetFileWriter]" args="[RunConfigMetoutFile]" />
>
> </dir>
>
> </output>
>
>
> After ingestion of this output product, I'd like to save the filename in
> my workflow metadata. How would I do this?
>
>
> I tried creating a PostIngestSuccess action, but found out that whatever
> metadata saving you want to do on the "crawler" side doesn't get passed
> back to the workflow metadata.
>
>
> I suppose I could update the workflow metadata by creating a
> PostIngestSuccess action and using the workflow manager client API to do
> this. However, when I look at this class, I see the following method,
>
>
> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>
>
> where it requires the workflow instance ID as a parameter. Is there a way
> to capture the workflow instance ID into the workflow metadata so that I
> can leverage this method to update my metadata?
>
>
> Thanks,
>
> Mike
>
>
>
>
--
-Sheryl
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On Mar 2, 2012, at 12:36 PM, Cayanan, Michael D (388J) wrote:
> Hi,
>
> In my PGE, I specify my output like the following:
>
> <output>
> <dir path="[ProductLocation]" createBeforeExe="true">
> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
> </dir>
> </output>
>
> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
You could extend CAS-PGE, and override the protected void runIngestCrawler(StdProductCrawler crawler,
List<File> crawlDirs) method like so:
public class SMAPPge extends StdPGETaskInstance{
@Override
protected void runIngestCrawler(StdProductCrawler crawler,
List<File> crawlDirs) {
super.runIngestCrawler(crawler, crawlDirs);
for(IngestStatus status: crawler.getIngestStatus()){
this.pgeMetadata.addDyanmicMetadata("IngestedFiles", status. getProduct().getName());
}
}
See this file: http://s.apache.org/ym
>
> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
That's by design. Crawler actions are meant to perform functionality "using" a read-only copy of the workflow met. They are *not* intended to
modify workflow met.
>
> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>
> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>
>
> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
You've got the workflow instance id. It's given to you by the workflow manager in the metadata context for each workflow instance, under the key name:
"WorkflowInstId"
See here: http://s.apache.org/NNs
and search for "getSharedContext()" for the list of keys that are automatically set.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by Sheryl John <sh...@gmail.com>.
Hi Mike,
You can use any of the Workflow Core met keys inside your PGE.
So, if you want to use the WorkflowInstId, you would specify it like this:
[WorkflowInstId] or [JobId] inside your PGE script.
But there could be other ways to update workflow metadata. You could also
have this in you metFileWriterClass, but that will be before your file is
ingested.
On Fri, Mar 2, 2012 at 12:36 PM, Cayanan, Michael D (388J) <
michael.d.cayanan@jpl.nasa.gov> wrote:
> Hi,
>
> In my PGE, I specify my output like the following:
>
> <output>
>
> <dir path="[ProductLocation]" createBeforeExe="true">
>
> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass=
> "[MetFileWriter]" args="[ConfigMetoutFile]" />
>
> <files name=
> "[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass=
> "[MetFileWriter]" args="[RunConfigMetoutFile]" />
>
> </dir>
>
> </output>
>
>
> After ingestion of this output product, I'd like to save the filename in
> my workflow metadata. How would I do this?
>
>
> I tried creating a PostIngestSuccess action, but found out that whatever
> metadata saving you want to do on the "crawler" side doesn't get passed
> back to the workflow metadata.
>
>
> I suppose I could update the workflow metadata by creating a
> PostIngestSuccess action and using the workflow manager client API to do
> this. However, when I look at this class, I see the following method,
>
>
> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>
>
> where it requires the workflow instance ID as a parameter. Is there a way
> to capture the workflow instance ID into the workflow metadata so that I
> can leverage this method to update my metadata?
>
>
> Thanks,
>
> Mike
>
>
>
>
--
-Sheryl
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, that makes sense.
Cheers,
Chris
On Mar 3, 2012, at 1:10 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> Ya actually you're right. After thinking about this more, I ended up doing what you mention below. Only I used the workflow instance id in the filemgr product metadata instead.
>
> It's much cleaner going this route (adding the job_id or workflow instance id to the flilemgr) instead of adding the filename to the workflow metadata.
>
> -Mike
>
> From: holenoter <ho...@mac.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Fri, 2 Mar 2012 13:23:51 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: PGE Question
>
> hey mike,
>
> why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
>
> -brian
>
> On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
>>
>> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
>>
>> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
>>
>> Thanks,
>> Mike
>>
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, that makes sense.
Cheers,
Chris
On Mar 3, 2012, at 1:10 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> Ya actually you're right. After thinking about this more, I ended up doing what you mention below. Only I used the workflow instance id in the filemgr product metadata instead.
>
> It's much cleaner going this route (adding the job_id or workflow instance id to the flilemgr) instead of adding the filename to the workflow metadata.
>
> -Mike
>
> From: holenoter <ho...@mac.com>
> Reply-To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Date: Fri, 2 Mar 2012 13:23:51 -0800
> To: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: PGE Question
>
> hey mike,
>
> why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
>
> -brian
>
> On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hi,
>>
>> In my PGE, I specify my output like the following:
>>
>> <output>
>> <dir path="[ProductLocation]" createBeforeExe="true">
>> <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
>> <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
>> </dir>
>> </output>
>>
>> After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
>>
>> I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
>>
>> I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
>>
>> updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
>>
>>
>> where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
>>
>> Thanks,
>> Mike
>>
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ya actually you're right. After thinking about this more, I ended up doing what you mention below. Only I used the workflow instance id in the filemgr product metadata instead.
It's much cleaner going this route (adding the job_id or workflow instance id to the flilemgr) instead of adding the filename to the workflow metadata.
-Mike
From: holenoter <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Fri, 2 Mar 2012 13:23:51 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: PGE Question
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ya actually you're right. After thinking about this more, I ended up doing what you mention below. Only I used the workflow instance id in the filemgr product metadata instead.
It's much cleaner going this route (adding the job_id or workflow instance id to the flilemgr) instead of adding the filename to the workflow metadata.
-Mike
From: holenoter <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Fri, 2 Mar 2012 13:23:51 -0800
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: PGE Question
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by holenoter <ho...@mac.com>.
job_id should be JobId
On Mar 02, 2012, at 01:23 PM, holenoter <ho...@mac.com> wrote:
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by holenoter <ho...@mac.com>.
job_id should be JobId
On Mar 02, 2012, at 01:23 PM, holenoter <ho...@mac.com> wrote:
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
� � � � � �<dir path="[ProductLocation]" createBeforeExe="true">
� � � � � � � <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]"�args="[ConfigMetoutFile]" />
� � � � � � � <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"�metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
� �</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method, � ��
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by holenoter <ho...@mac.com>.
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
<dir path="[ProductLocation]" createBeforeExe="true">
<files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]" args="[ConfigMetoutFile]" />
<files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml" metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method,
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike
Re: PGE Question
Posted by holenoter <ho...@mac.com>.
hey mike,
why do you need the filename in the workflow metadata?... you can add the job_id to your filemgr product metadata and then you can just query the filemgr based on job_id to find out which files where created for that job.
-brian
On Mar 02, 2012, at 12:36 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi,
In my PGE, I specify my output like the following:
<output>
� � � � � �<dir path="[ProductLocation]" createBeforeExe="true">
� � � � � � � <files regExp="SMAP_L0B_RADAR_SPS.*\.h5" metFileWriterClass="[MetFileWriter]"�args="[ConfigMetoutFile]" />
� � � � � � � <files name="[PGETask_Name]_RunConfig_[ProductionDateTime].xml"�metFileWriterClass="[MetFileWriter]" args="[RunConfigMetoutFile]" />
� �</dir>
</output>
After ingestion of this output product, I'd like to save the filename in my workflow metadata. How would I do this?
I tried creating a PostIngestSuccess action, but found out that whatever metadata saving you want to do on the "crawler" side doesn't get passed back to the workflow metadata.
I suppose I could update the workflow metadata by creating a PostIngestSuccess action and using the workflow manager client API to do this. However, when I look at this class, I see the following method, � ��
updateMetadataForWorkflow(String workflowInstId, Metadata metadata)
where it requires the workflow instance ID as a parameter. Is there a way to capture the workflow instance ID into the workflow metadata so that I can leverage this method to update my metadata?
Thanks,
Mike