You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mallder, Valerie" <Va...@jhuapl.edu> on 2014/10/15 23:33:15 UTC
SQL parsing error in PgeConfig
Hi Chris,
I played with the SQL statement in my pge config file today and keep getting a parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you have a cut-and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if I understand correctly what you are doing there. And this typo propagated its way to the other xml files as well), and I have tried to mimic the parts that apply to what I am trying to do and I'm stumped by the SQL parsing error.
I am doing a very simple query just to try to prove that I *can* do a query. I didn't really need the FORMAT agreement, but I wasn't sure the right syntax for leaving it out. So I just specified the metadata key I'm searching for in the FORMAT arguement. So here's what I have.
<metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }"/>
One difference between this statement and yours is that you used $ signs in your FORMAT specification. I have tried it both ways, with dollar signs and without dollar signs and still get the parse error. (What is the significance of the "$"?)
Here's the output from oodt.out, and my pge config file and tasks.xml follow afterwards. The stack trace looks familiar from the standpoint that I've seen parse errors before and fixed them. But, this one points out a suspicious looking exception for a malformed URL for some reason. However, the output from the log file indicates all of my urls are fine. No pge metadata is written for this task and no script is created. Please let me know if you see anything obvious that I'm missing.
Thanks,
Val
Oct 15, 2014 5:15:02 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread checkTaskRequiredMetadata
INFO: Task: [DB-Builder-Task] has no required metadata fields
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread executeTaskLocally
INFO: Executing task: [DB-Builder-Task] locally
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
INFO: Converting workflow configuration to static metadata...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_CrawlerConfigFile] value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler/policy/crawler-config.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value = [http://localhost:9001]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl] value = [http://localhost:9000]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [FEI_DROP_DIR] value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGE_HOME] value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Query_ClientTransferServiceFactory] value = [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value = [MoveFileToLevel0Dir]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo] value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/policy/mime-extractor-map.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value = [true]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl] value = [http://localhost:9000]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_Name] value = [DB-Builder-Task]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/config/jedi-conduit-db-builder-config.xml]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
INFO: Loading workflow context metadata...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = [44f791c9-54b0-11e4-a032-c9e22d0219d9]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding dynamic metadata: key = [JobId] value = [44f791c9-54b0-11e4-a032-c9e22d0219d9]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value = [http://slothrop.jhuapl.edu:9001]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding dynamic metadata: key = [TaskId] value = [urn:oodt:dbBuilderTask]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
FINEST: Adding dynamic metadata: key = [ProcessingNode] value = [slothrop.jhuapl.edu]
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
INFO: Create PgeConfig...
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
INFO: Using default PgeConfigBuilder: org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run
SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
java.io.IOException: Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:230)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.Exception: Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
at org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(XmlFilePgeConfigBuilder.java:165)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFilePgeConfigBuilder.java:120)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:83)
... 6 more
Caused by: java.net.MalformedURLException
at java.net.URL.<init>(URL.java:619)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
... 11 more
Caused by: java.lang.NullPointerException
at java.net.URL.<init>(URL.java:524)
... 14 more
org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeTaskLocally(IterativeWorkflowProcessorThread.java:574)
at org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(IterativeWorkflowProcessorThread.java:321)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:87)
at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:230)
at org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
... 4 more
Caused by: java.lang.Exception: Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
at org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(XmlFilePgeConfigBuilder.java:165)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFilePgeConfigBuilder.java:120)
at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:83)
... 6 more
Caused by: java.net.MalformedURLException
at java.net.URL.<init>(URL.java:619)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
... 11 more
Caused by: java.lang.NullPointerException
at java.net.URL.<init>(URL.java:524)
... 14 more
Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread executeTaskLocally
WARNING: Exception executing task: [DB-Builder-Task] locally: Message: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value: SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
<!--
** Filename: jedi-conduit-db-builder-config.xml
-->
<pgeConfig>
<import file="pge-common-config.xml" />
<!-- Note: do not use the "shell" attribute, "shellType is the correct attribute here"-->
<exe dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/pge/jobs" shellType="/usr/local/bin/bash">
<cmd>echo "Current Working Directory is `pwd`" > [JobLogDir]/db-builder.log</cmd>
<cmd>echo "Current Environment is `/bin/env`" >> [JobLogDir]/db-builder.log</cmd>
<cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList] [Arguements]</cmd>
</exe>
<!-- Custom metadata to add to output files -->
<customMetadata>
<metadata key="LoadClass" val="conduit.jedi.utils.JEDIFastDBBuilder" />
<metadata key="TelemetryFileDb" val="[JobDir]/valsTelemetryFile.db" />
<metadata key="Arguements" val="[TelemetryFileDb]" />
<metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }" />
<!-- SQL example statement from the code
SQL (FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime,EndDateTime,DataVersion,TakeHighestPriority) { SELECT FileLocation,Filename,FileSize FROM IASI_L1C WHERE ProductionDateTime >= '2007-12-01T00:00:00.000000Z' }
-->
</customMetadata>
</pgeConfig>
<!--
** Filename: tasks.xml
-->
<task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task" class="org.apache.oodt.cas.pge.StdPGETaskInstance">
<conditions/>
<configuration>
<property name="PGETask_Name" value="DB-Builder-Task"/>
<property name="PGETask_ConfigFilePath" value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/config/jedi-conduit-db-builder-config.xml" envReplace="true" />
<property name="PGETask_DumpMetadata" value="true"/>
<property name="PCS_WorkflowManagerUrl" value="http://localhost:9001" />
<property name="PGETask_Query_FileManagerUrl" value="http://localhost:9000" />
<property name="PGETask_Ingest_FileManagerUrl" value="http://localhost:9000"/>
<property name="PGETask_Query_ClientTransferServiceFactory" value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/>
<property name="PGETask_Ingest_CrawlerConfigFile" value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/crawler/policy/crawler-config.xml"/>
<property name="PGETask_Ingest_MimeExtractorRepo" value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/extensions/policy/mime-extractor-map.xml"/>
<property name="PGETask_Ingest_ActionIds" value="MoveFileToLevel0Dir"/>
<property name="PGE_HOME" value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/>
<property name="FEI_DROP_DIR" value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging"/>
</configuration>
<requiredMetFields/>
</task>
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Chris Mattmann [mailto:chris.mattmann@gmail.com]
> Sent: Tuesday, October 14, 2014 4:11 PM
> To: dev@oodt.apache.org
> Subject: Re: Failed to build PgeConfig, exception in
> PathUtils.doDynamicReplacement
>
> Awesome Val, glad it helped and that you are moving further!
>
> Here?s a quick example of how to call your perl script as an extractor in CAS-PGE
> (barring yes, the duplication of specifying a regExp to select the type to run the
> extractor on, but I?ll get to that later):
>
> Under <output>:
>
> <files regExp=?\\*.hdf"
>
> metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorMetWri
> ter"
> args=?/path/to/your-extern-config.xml?>
>
>
> Your extern-config.xml file should work like this:
>
> http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources/exam
> ples/extern-config.xml
>
>
> Note that the ExternExtractor is documented here:
>
> http://oodt.apache.org/components/maven/metadata/user/basic.html
>
>
> That will get you your Perl script integrated as an extractor into the system.
>
> On to your question about selecting information from the File Manager:
>
> Look up the SQL(.. portions of the DRAT PGE configs to get an idea of how to do
> this. In short, yes you can do it, and the way to do it is to use the special CAS-
> PGE SQL-like language that you can prefix your met key with (make sure
> envReplace is turned on). You can do like SQL([options]){ SELECT FROM
> <product type name> WHERE <criteria>
>
> The results of your above query are stored into the metadata field name in your
> CAS-PGE config file.
>
> I?ll walk you through more of this later, but have to run to a meeting.
>
> Gluck!
>
> Cheers,
> Chris
>
>
> ------------------------
> Chris Mattmann
> chris.mattmann@gmail.com
>
>
>
>
> -----Original Message-----
> From: "Mallder, Valerie" <Va...@jhuapl.edu>
> Reply-To: <de...@oodt.apache.org>
> Date: Tuesday, October 14, 2014 at 12:58 PM
> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> Subject: RE: Failed to build PgeConfig, exception in
> PathUtils.doDynamicReplacement
>
> >Hi Chris,
> >
> >Thanks for your emails from Friday and Saturday. They were extremely
> >helpful and I have now worked my way through all of my setup issues so
> >far. I was using the new style CASE-PGE keys because Rishi used those
> >in his working example and at one point I had been trying to mimic that
> >example. I didn't know anything about the legacy mode flag. But, after
> >changing the keys to the old style, everything started to come together
> >a little faster.
> >
> >To start my crawler at the beginning of my pipeline, I chose the
> >approach that Rishi had taken where the pgeConfig starts the
> >crawler_launcher script directly rather than rely on the definition of
> >the output files to trigger the crawler to run. This was because the
> >met extractor I am using is a perl script, and your DRAT system only
> >shows how to specify a class for the metFileWriterClass attribute. I
> >could not find anything to show me how to specify a perl script as the met file
> writer instead.
> >Also, having to specify the file types to crawl for by using a regular
> >expression in the config file seemed like an unnecessary duplication of
> >the definition of the files types to crawl for (since the regular
> >expression just specifies a mime-type that is already defined in
> >mime-types.xml). I didn't want to specify the mime-type in two
> >different places.
> >
> >So, now I've been through the process of setting up my CAS-PGE to
> >ingest the raw science and spacecraft housekeeping telemetry files, and
> >run a postIngestSuccess action to copy some of those files to a new
> >directory for further processing.
> >
> >The next step in my pipeline may be a little tricky. I need to query
> >the filemgr and pass the results of the query (it could be a long list)
> >as an input parameter to the first algorithm in the pipeline which,
> >happens to be another perl script. Does anyone know how I might do
> >this in a PGE config file?
> >
> >Thanks!!
> >Val
> >
> >
> >
> >
> >
> >Valerie A. Mallder
> >New Horizons Deputy Mission System Engineer Johns Hopkins
> >University/Applied Physics Laboratory
> >
> >
> >> -----Original Message-----
> >> From: Mattmann, Chris A (3980) [mailto:chris.a.mattmann@jpl.nasa.gov]
> >> Sent: Friday, October 10, 2014 7:59 PM
> >> To: dev@oodt.apache.org
> >> Subject: Re: Failed to build PgeConfig, exception in
> >> PathUtils.doDynamicReplacement
> >>
> >> Thanks Val you are close!
> >>
> >> Looking at what you show below, I think the issue is one of the
> >>following:
> >>
> >> 1. You are using the new style CAS-PGE Keys. Try using the old ones
> >>(e.g., the ones present here:
> >>
> >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/exam
> >>ple
> >>s/
> >> WorkflowTask/tasks.xml
> >>
> >> (note the ?_? and not the ?/? used in keys).
> >> These keys are used by CAS-PGE if you see something in your wmgr bin
> >>script (or your resource manager batch_stub script) stating
> >>?legacyMode=true?).
> >> If you
> >> are using RADIX, I believe that?s the case:
> >>
> >> http://s.apache.org/hO
> >>
> >> 2. Regarding better building environments. The m2e plugin for Eclipse
> >>is fantastic nowadays and can literally checkout a multi-module Maven
> >>project from SVN (when paired with Subversive as a plugin or
> >>Subclipse). Another thing to check out is this page on the wiki:
> >>
> >>
> >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develope
> >>r+H
> >>el
> >> p
> >>
> >>
> >> Let me know if that fixes it. Sorry for all the trouble :) Trust me
> >>it will be worth it.
> >> CAS-PGE is awesome once working.
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >> Chris Mattmann, Ph.D.
> >> Chief Architect
> >> Instrument Software and Science Data Systems Section (398) NASA Jet
> >> Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 168-519, Mailstop: 168-527
> >> Email: chris.a.mattmann@nasa.gov
> >> WWW: http://sunset.usc.edu/~mattmann/
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >> Adjunct Associate Professor, Computer Science Department University
> >> of Southern California, Los Angeles, CA 90089 USA
> >>
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> +++++
> >>
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: <Mallder>, Valerie <Va...@jhuapl.edu>
> >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> Date: Friday, October 10, 2014 at 3:40 PM
> >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> >> Subject: Failed to build PgeConfig, exception in
> >>PathUtils.doDynamicReplacement
> >>
> >> >Hi All,
> >> >
> >> >I'm still working on this! Still trying to get a CA-PGE Task to run.
> >> >I'm almost there. Now it at least trys to build the PGE config file.
> >>But the
> >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a
> >> >null string somewhere, but it also looks like it is crashing in a
> >> >strange place. It is unable to read the .xml file that contains my
> >> ><pgeConfig>...</pgeConfig> but, the correct path and file are shown
> >> >in the output log messages I've include below which also shows the
> >> >stack trace. I looks like it is failing during a recursive call to
> >> >PathUtils.doDynamicReplacement. I saw some chatter about some prior
> >> >errors in this code in the mailing list archives from 2010 and 2012.
> >> >So I am hoping someone might remember and be able to tell me if I am
> >> >doing something that is causing this error. I included my tasks.xml
> >> >and fei-crawler-pge-config.xml file after the log messages.
> >> >
> >> >At this point, I think need to start running these processes in an
> >> >environment where I can debug this better. The runtime output and
> >> >stack traces simply aren't enough for me to track this down. I am
> >> >newbie to using eclipse and mvn, so, does anyone have some notes on
> >> >the best way to import oodt-0.7 into an exclipe workspace and tell
> >> >it to build it using mvn?? I am hoping I can leverage other peoples
> >> >knowledge of how to do this so that I can do it quickly and not
> >> >waste another week. I imported the oodt-0.7 directory into eclipse
> >> >so I could view the files easier, but I just made eclipse link to
> >> >the where the files are located rather that copy them into a
> >> >workspace folder. If there's a better way, please tell me. And now,
> >> >how do I tell exclipse
> >>to build
> >> stuff?
> >> >
> >> >Thanks!!
> >> >Valerie
> >> >
> >> >
> >> >
> >> >Using CATALINA_BASE:
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >> >Using CATALINA_HOME:
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> >> >Using CATALINA_TMPDIR:
> >>
> >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat/t
> >>>emp
> >> >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55
> >> >Workflow Manager started PID file
> >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl
> >> >ow/
> >> >run
> >> >/cas.workflow.pid).
> >> >Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >loadProperties
> >> >INFO: Loading Workflow Manager Configuration Properties from:
> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/workfl
> >> >ow/
> >> >etc
> >> >/workflow.properties]
> >> >Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactory
> >> >getResmgrUrl
> >> >INFO: No Resource Manager URL provided or malformed URL: executing
> >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
> >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM
> >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >handleEvent
> >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10,
> >> >2014
> >> >5:54:12 PM org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> >> >handleEvent
> >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for event
> >> >startJediPipeline Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >checkTaskRequiredMetadata
> >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields Oct
> >> >10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >executeTaskLocally
> >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014
> >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >INFO: Converting workflow configuration to static metadata...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl]
> >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/CrawlerConfigFile]
> >> >value =
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/c
> >> >raw
> >> >ler
> >> >/policy/crawler-config.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Name] value =
> >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds]
> >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value =
> >> >[true] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Query/ClientTransferServiceFactory] value =
> >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGE_HOME] value =
> >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/Query/FileManagerUrl]
> >> >value = [http://localhost:9000] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/MimeExtractorRepo]
> >> >value =
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
> >> >xte
> >> >nsi
> >> >ons/policy/mime-extractor-map.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath] value
> >> >=
> >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/e
> >> >xte
> >> >nsi
> >> >ons/config/fei-crawler-pge-config.xml]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding static metadata: key =
> >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000] Oct
> >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >INFO: Loading workflow context metadata...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [JobId] value =
> >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
> >> >[http://slothrop.jhuapl.edu:9001]
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [TaskId] value =
> >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeMetadata
> >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
> >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeConfig
> >> >INFO: Create PgeConfig...
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> >> >createPgeConfig
> >> >INFO: Using default PgeConfigBuilder:
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
> >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance run
> >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
> >> >parse
> >> >value: null
> >> >java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> >> >null
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:87)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta
> >> >nce
> >> >.ja
> >> >va:230)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ex
> >> >ecu
> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ru
> >> >n(I
> >> >terativeWorkflowProcessorThread.java:321)
> >> > at
> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> >>Source)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >Caused by: java.lang.Exception: Failed to parse value: null
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:77)
> >> > ... 6 more
> >> >Caused by: java.lang.NullPointerException
> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep
> >> >lac
> >> >eme
> >> >nt(PathUtils.java:321)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat
> >> >hUt
> >> >ils
> >> >.java:96)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >> > ... 8 more
> >> >
> >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstance
> >> >Exc
> >> >ept
> >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
> >> >value: null
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ex
> >> >ecu
> >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> >> > at
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >.ru
> >> >n(I
> >> >terativeWorkflowProcessorThread.java:321)
> >> > at
> >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> >>Source)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed
> >> >to parse value: null
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:87)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInsta
> >> >nce
> >> >.ja
> >> >va:230)
> >> > at
> >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> >> > ... 4 more
> >> >Caused by: java.lang.Exception: Failed to parse value: null
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> >> > at
> >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFile
> >> >Pge
> >> >Con
> >> >figBuilder.java:77)
> >> > ... 6 more
> >> >Caused by: java.lang.NullPointerException
> >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisRep
> >> >lac
> >> >eme
> >> >nt(PathUtils.java:321)
> >> > at
> >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(Pat
> >> >hUt
> >> >ils
> >> >.java:96)
> >> > at
> >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> >> > ... 8 more
> >> >Oct 10, 2014 5:54:13 PM
> >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> >> >executeTaskLocally
> >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally:
> >>Message:
> >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> >> >null
> >> >
> >> >
> >> >
> >> >
> >> >Content of tasks.xml
> >> >
> >> >
> >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
> >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
> >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance">
> >> > <conditions/>
> >> >
> >> > <configuration>
> >> > <property name="PGETask/Name" value="feiCrawlerTaskName"/>
> >> > <property name="PGETask/ConfigFilePath"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo y/e xtensions/config/fei-crawler-pge-config.xml"/>
> >> > <property name="PGETask/DumpMetadata" value="true"/>
> >> > <property name="PGETask/WorkflowManagerUrl"
> >> >value="http://localhost:9001" />
> >> > <property name="PGETask/Query/FileManagerUrl"
> >> >value="http://localhost:9000" />
> >> > <property name="PGETask/Ingest/FileManagerUrl"
> >> >value="http://localhost:9000"/>
> >> >
> >> > <property name="PGETask/Query/ClientTransferServiceFactory"
> >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFac
> >> >tor
> >> >y"/
> >> >>
> >> > <property name="PGETask/Ingest/CrawlerConfigFile"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo
> >> >y/c
> >> >rawler/policy/crawler-config.xml"/>
> >> > <property name="PGETask/Ingest/MimeExtractorRepo"
> >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-de
> >> >plo y/e xtensions/policy/mime-extractor-map.xml"/>
> >> > <property name="PGETask/Ingest/ActionIds"
> >> >value="MoveFileToLevel0Dir"/>
> >> > <property name="PGE_HOME"
> >>
> >>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"
> >>>/>
> >> > </configuration>
> >> >
> >> > <requiredMetFields/>
> >> > </task>
> >> ></cas:tasks>
> >> >
> >> >
> >> >Contents of fei-crawler-pge-config.xml
> >> >
> >> >
> >> ><pgeConfig>
> >> >
> >> > <!-- How to run the PGE -->
> >> > <!-just echoing the current directory to a file so I can see if
> >> >this thing ever starts -->
> >> > <exe
> >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/da
> >> >ta/
> >> >pge
> >> >/jobs" shell="/bin/sh">
> >> > <cmd>echo "Current Working Directory is `pwd`" >
> >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/pg
> >> >e.l
> >> >og<
> >> >/cmd>
> >> > </exe>
> >> >
> >> > <!-- Files to ingest -->
> >> > <output>
> >> > <!-- trying this approach. Telling PGE there is output files
> >> >should invoke the crawler. -->
> >> >
> >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/d
> >> >ata
> >> >/st
> >> >aging"/>
> >> > </output>
> >> >
> >> > <!-- Custom metadata to add to output files -->
> >> > <customMetadata>
> >> >
> >> > <!-- each of these directories exist -->
> >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
> >> >envReplace="true"/>
> >> > <metadata key="JobInputDir" val="[JobDir]/input"/>
> >> > <metadata key="JobOutputDir" val="[JobDir]/output"/>
> >> > <metadata key="JobLogDir" val="[JobDir]/logs"/>
> >> >
> >> > </customMetadata>
> >> ></pgeConfig>
> >> >
> >> >Valerie A. Mallder
> >> >
> >> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> >> >University/Applied Physics Laboratory
> >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> >> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> >> >
> >
>
Re: SQL parsing error in PgeConfig
Posted by Chris Mattmann <ch...@gmail.com>.
Not stupid at all, an honest mistake that we need better
error logging to inform users of.
Thanks for sticking with it, Val.
Cheers,
Chris
------------------------
Chris Mattmann
chris.mattmann@gmail.com
-----Original Message-----
From: "Mallder, Valerie" <Va...@jhuapl.edu>
Reply-To: <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 11:10 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: RE: SQL parsing error in PgeConfig
>I was not setting the property "PCS_FileManagerUrl" to the [FILEMGR_URL]
>in the tasks.xml file. Instead, I was setting a property name that
>doesn't exist called "PGETask_Query_FileManagerUrl" to [FILEMGR_URL] .
>
>This is what happens when you run into so many different types of
>problems. You forget what you changed, and why you change it, and so you
>don't go back and review all the changes to verify they are correct. I
>thought I knew the correct name of the key, and so I didn't check it
>against the code, and it was wrong. Stupid, stupid, stupid.
>
>
>Valerie A. Mallder
>New Horizons Deputy Mission System Engineer
>Johns Hopkins University/Applied Physics Laboratory
>
>
>> -----Original Message-----
>> From: Tyler Palsulich [mailto:tpalsulich@gmail.com]
>> Sent: Thursday, October 16, 2014 1:56 PM
>> To: dev
>> Subject: Re: SQL parsing error in PgeConfig
>>
>> > Found it!
>>
>> Woo! What was wrong?
>>
>> Tyler
Re: SQL parsing error in PgeConfig
Posted by Tyler Palsulich <tp...@gmail.com>.
Ah. Sounds like an easy mistake. Thanks for the heads up!
Tyler
On Thu, Oct 16, 2014 at 11:10 AM, Mallder, Valerie <
Valerie.Mallder@jhuapl.edu> wrote:
> I was not setting the property "PCS_FileManagerUrl" to the [FILEMGR_URL]
> in the tasks.xml file. Instead, I was setting a property name that doesn't
> exist called "PGETask_Query_FileManagerUrl" to [FILEMGR_URL] .
>
> This is what happens when you run into so many different types of
> problems. You forget what you changed, and why you change it, and so you
> don't go back and review all the changes to verify they are correct. I
> thought I knew the correct name of the key, and so I didn't check it
> against the code, and it was wrong. Stupid, stupid, stupid.
>
>
> Valerie A. Mallder
> New Horizons Deputy Mission System Engineer
> Johns Hopkins University/Applied Physics Laboratory
>
>
> > -----Original Message-----
> > From: Tyler Palsulich [mailto:tpalsulich@gmail.com]
> > Sent: Thursday, October 16, 2014 1:56 PM
> > To: dev
> > Subject: Re: SQL parsing error in PgeConfig
> >
> > > Found it!
> >
> > Woo! What was wrong?
> >
> > Tyler
>
RE: SQL parsing error in PgeConfig
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
I was not setting the property "PCS_FileManagerUrl" to the [FILEMGR_URL] in the tasks.xml file. Instead, I was setting a property name that doesn't exist called "PGETask_Query_FileManagerUrl" to [FILEMGR_URL] .
This is what happens when you run into so many different types of problems. You forget what you changed, and why you change it, and so you don't go back and review all the changes to verify they are correct. I thought I knew the correct name of the key, and so I didn't check it against the code, and it was wrong. Stupid, stupid, stupid.
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Tyler Palsulich [mailto:tpalsulich@gmail.com]
> Sent: Thursday, October 16, 2014 1:56 PM
> To: dev
> Subject: Re: SQL parsing error in PgeConfig
>
> > Found it!
>
> Woo! What was wrong?
>
> Tyler
Re: SQL parsing error in PgeConfig
Posted by Tyler Palsulich <tp...@gmail.com>.
> Found it!
Woo! What was wrong?
Tyler
Re: SQL parsing error in PgeConfig
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
You are definitely not an idiot - you are awesome! :) Great work.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: <Mallder>, Valerie <Va...@jhuapl.edu>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Thursday, October 16, 2014 at 9:07 AM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: RE: SQL parsing error in PgeConfig
>Found it! Ugh! I'm an idiot! Got the sql to work. Now moving on to
>figuring out why the class in my jar file can't be found. But that's
>progress! Yay!
>
>
>Valerie A. Mallder
>New Horizons Deputy Mission System Engineer
>Johns Hopkins University/Applied Physics Laboratory
>
>
>> -----Original Message-----
>> From: Mallder, Valerie [mailto:Valerie.Mallder@jhuapl.edu]
>> Sent: Wednesday, October 15, 2014 5:33 PM
>> To: dev@oodt.apache.org
>> Subject: SQL parsing error in PgeConfig
>>
>> Hi Chris,
>>
>> I played with the SQL statement in my pge config file today and keep
>>getting a
>> parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you
>>have a cut-
>> and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if
>>I understand
>> correctly what you are doing there. And this typo propagated its way to
>>the other
>> xml files as well), and I have tried to mimic the parts that apply to
>>what I am trying
>> to do and I'm stumped by the SQL parsing error.
>>
>> I am doing a very simple query just to try to prove that I *can* do a
>>query. I didn't
>> really need the FORMAT agreement, but I wasn't sure the right syntax
>>for leaving it
>> out. So I just specified the metadata key I'm searching for in the
>>FORMAT
>> arguement. So here's what I have.
>>
>> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT
>>FileLocation
>> FROM EngineeringFile }"/>
>>
>> One difference between this statement and yours is that you used $
>>signs in your
>> FORMAT specification. I have tried it both ways, with dollar signs and
>>without
>> dollar signs and still get the parse error. (What is the significance
>>of the "$"?)
>>
>> Here's the output from oodt.out, and my pge config file and tasks.xml
>>follow
>> afterwards. The stack trace looks familiar from the standpoint that
>>I've seen parse
>> errors before and fixed them. But, this one points out a suspicious
>>looking
>> exception for a malformed URL for some reason. However, the output from
>>the log
>> file indicates all of my urls are fine. No pge metadata is written for
>>this task and no
>> script is created. Please let me know if you see anything obvious that
>>I'm missing.
>>
>> Thanks,
>> Val
>>
>>
>>
>> Oct 15, 2014 5:15:02 PM
>> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>> checkTaskRequiredMetadata
>> INFO: Task: [DB-Builder-Task] has no required metadata fields Oct 15,
>>2014
>> 5:15:03 PM
>> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>> executeTaskLocally
>> INFO: Executing task: [DB-Builder-Task] locally Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> INFO: Converting workflow configuration to static metadata...
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key =
>>[PGETask_Ingest_CrawlerConfigFile] value
>> = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/crawler/policy/crawler-config.xml]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value =
>> [http://localhost:9001] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl]
>>value =
>> [http://localhost:9000] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding static metadata: key = [FEI_DROP_DIR] value =
>>
>>[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/stagin
>>g]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PGE_HOME] value =
>> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key =
>> [PGETask_Query_ClientTransferServiceFactory] value =
>> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value =
>> [MoveFileToLevel0Dir] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo]
>> value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/extensions/policy/mime-extractor-map.xml]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value =
>>[true]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl]
>>value =
>> [http://localhost:9000] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_Name] value =
>>[DB-Builder-
>> Task] Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value =
>> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/extensions/config/jedi-conduit-db-builder-config.xml]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> INFO: Loading workflow context metadata...
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
>>[44f791c9-
>> 54b0-11e4-a032-c9e22d0219d9]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding dynamic metadata: key = [JobId] value =
>>[44f791c9-54b0-11e4-
>> a032-c9e22d0219d9]
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeMetadata
>> FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
>> [http://slothrop.jhuapl.edu:9001] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding dynamic metadata: key = [TaskId] value =
>> [urn:oodt:dbBuilderTask] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
>> FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
>> [slothrop.jhuapl.edu] Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
>> INFO: Create PgeConfig...
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
>> createPgeConfig
>> INFO: Using default PgeConfigBuilder:
>> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
>> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run
>> SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
>>value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> java.io.IOException: Failed to build PgeConfig : Failed to parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo
>>nfig
>> Builder.java:87)
>> at
>>
>>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.j
>>a
>> va:230)
>> at
>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>> at
>>
>>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.exec
>>uteT
>> askLocally(IterativeWorkflowProcessorThread.java:574)
>> at
>>
>>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(
>>Iterati
>> veWorkflowProcessorThread.java:321)
>> at
>>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.lang.Exception: Failed to parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>> at
>>
>>org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:2
>>45)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata
>>(X
>> mlFilePgeConfigBuilder.java:165)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(Xml
>>File
>> PgeConfigBuilder.java:120)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo
>>nfig
>> Builder.java:83)
>> ... 6 more
>> Caused by: java.net.MalformedURLException
>> at java.net.URL.<init>(URL.java:619)
>> at java.net.URL.<init>(URL.java:482)
>> at java.net.URL.<init>(URL.java:431)
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
>> ... 11 more
>> Caused by: java.lang.NullPointerException
>> at java.net.URL.<init>(URL.java:524)
>> ... 14 more
>>
>>
>>org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceExcep
>>tion
>> : PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> at
>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
>> at
>>
>>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.exec
>>uteT
>> askLocally(IterativeWorkflowProcessorThread.java:574)
>> at
>>
>>org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(
>>Iterati
>> veWorkflowProcessorThread.java:321)
>> at
>>EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: java.io.IOException: Failed to build PgeConfig : Failed to
>>parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo
>>nfig
>> Builder.java:87)
>> at
>>
>>org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.j
>>a
>> va:230)
>> at
>> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>> ... 4 more
>> Caused by: java.lang.Exception: Failed to parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>> at
>>
>>org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:2
>>45)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata
>>(X
>> mlFilePgeConfigBuilder.java:165)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(Xml
>>File
>> PgeConfigBuilder.java:120)
>> at
>>
>>org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeCo
>>nfig
>> Builder.java:83)
>> ... 6 more
>> Caused by: java.net.MalformedURLException
>> at java.net.URL.<init>(URL.java:619)
>> at java.net.URL.<init>(URL.java:482)
>> at java.net.URL.<init>(URL.java:431)
>> at
>>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
>> ... 11 more
>> Caused by: java.lang.NullPointerException
>> at java.net.URL.<init>(URL.java:524)
>> ... 14 more
>> Oct 15, 2014 5:15:03 PM
>> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
>> executeTaskLocally
>> WARNING: Exception executing task: [DB-Builder-Task] locally: Message:
>> PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
>> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>>
>> <!--
>> ** Filename: jedi-conduit-db-builder-config.xml
>> -->
>> <pgeConfig>
>>
>> <import file="pge-common-config.xml" />
>>
>> <!-- Note: do not use the "shell" attribute, "shellType is the
>>correct attribute here"-
>> ->
>> <exe dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/data/pge/jobs" shellType="/usr/local/bin/bash">
>> <cmd>echo "Current Working Directory is `pwd`" > [JobLogDir]/db-
>> builder.log</cmd>
>> <cmd>echo "Current Environment is `/bin/env`" >> [JobLogDir]/db-
>> builder.log</cmd>
>>
>> <cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList]
>> [Arguements]</cmd>
>> </exe>
>>
>> <!-- Custom metadata to add to output files -->
>> <customMetadata>
>> <metadata key="LoadClass"
>>val="conduit.jedi.utils.JEDIFastDBBuilder" />
>> <metadata key="TelemetryFileDb"
>>val="[JobDir]/valsTelemetryFile.db" />
>> <metadata key="Arguements" val="[TelemetryFileDb]" />
>>
>> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT
>> FileLocation FROM EngineeringFile }" />
>>
>> <!-- SQL example statement from the code
>> SQL
>>
>>(FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime,
>>En
>> dDateTime,DataVersion,TakeHighestPriority) { SELECT
>> FileLocation,Filename,FileSize FROM IASI_L1C WHERE ProductionDateTime
>>>=
>> '2007-12-01T00:00:00.000000Z' }
>> -->
>> </customMetadata>
>> </pgeConfig>
>>
>>
>> <!--
>> ** Filename: tasks.xml
>> -->
>>
>> <task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task"
>> class="org.apache.oodt.cas.pge.StdPGETaskInstance">
>> <conditions/>
>>
>> <configuration>
>> <property name="PGETask_Name" value="DB-Builder-Task"/>
>> <property name="PGETask_ConfigFilePath"
>> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/extensions/config/jedi-conduit-db-builder-config.xml"
>>envReplace="true" />
>> <property name="PGETask_DumpMetadata" value="true"/>
>> <property name="PCS_WorkflowManagerUrl"
>>value="http://localhost:9001"
>> />
>> <property name="PGETask_Query_FileManagerUrl"
>> value="http://localhost:9000" />
>> <property name="PGETask_Ingest_FileManagerUrl"
>> value="http://localhost:9000"/>
>>
>> <property name="PGETask_Query_ClientTransferServiceFactory"
>>
>>value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"
>>/>
>> <property name="PGETask_Ingest_CrawlerConfigFile"
>> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/crawler/policy/crawler-config.xml"/>
>> <property name="PGETask_Ingest_MimeExtractorRepo"
>> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/extensions/policy/mime-extractor-map.xml"/>
>> <property name="PGETask_Ingest_ActionIds"
>> value="MoveFileToLevel0Dir"/>
>> <property name="PGE_HOME"
>>
>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/
>>>
>>
>> <property name="FEI_DROP_DIR"
>> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> deploy/data/staging"/>
>> </configuration>
>> <requiredMetFields/>
>> </task>
>>
>>
>> Valerie A. Mallder
>> New Horizons Deputy Mission System Engineer Johns Hopkins
>>University/Applied
>> Physics Laboratory
>>
>> > -----Original Message-----
>> > From: Chris Mattmann [mailto:chris.mattmann@gmail.com]
>> > Sent: Tuesday, October 14, 2014 4:11 PM
>> > To: dev@oodt.apache.org
>> > Subject: Re: Failed to build PgeConfig, exception in
>> > PathUtils.doDynamicReplacement
>> >
>> > Awesome Val, glad it helped and that you are moving further!
>> >
>> > Here?s a quick example of how to call your perl script as an extractor
>> > in CAS-PGE (barring yes, the duplication of specifying a regExp to
>> > select the type to run the extractor on, but I?ll get to that later):
>> >
>> > Under <output>:
>> >
>> > <files regExp=?\\*.hdf"
>> >
>> > metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorM
>> > etWri
>> > ter"
>> > args=?/path/to/your-extern-config.xml?>
>> >
>> >
>> > Your extern-config.xml file should work like this:
>> >
>> > http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources
>> > /exam
>> > ples/extern-config.xml
>> >
>> >
>> > Note that the ExternExtractor is documented here:
>> >
>> > http://oodt.apache.org/components/maven/metadata/user/basic.html
>> >
>> >
>> > That will get you your Perl script integrated as an extractor into
>>the system.
>> >
>> > On to your question about selecting information from the File Manager:
>> >
>> > Look up the SQL(.. portions of the DRAT PGE configs to get an idea of
>> > how to do this. In short, yes you can do it, and the way to do it is
>> > to use the special CAS- PGE SQL-like language that you can prefix your
>> > met key with (make sure envReplace is turned on). You can do like
>> > SQL([options]){ SELECT FROM <product type name> WHERE <criteria>
>> >
>> > The results of your above query are stored into the metadata field
>> > name in your CAS-PGE config file.
>> >
>> > I?ll walk you through more of this later, but have to run to a
>>meeting.
>> >
>> > Gluck!
>> >
>> > Cheers,
>> > Chris
>> >
>> >
>> > ------------------------
>> > Chris Mattmann
>> > chris.mattmann@gmail.com
>> >
>> >
>> >
>> >
>> > -----Original Message-----
>> > From: "Mallder, Valerie" <Va...@jhuapl.edu>
>> > Reply-To: <de...@oodt.apache.org>
>> > Date: Tuesday, October 14, 2014 at 12:58 PM
>> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > Subject: RE: Failed to build PgeConfig, exception in
>> > PathUtils.doDynamicReplacement
>> >
>> > >Hi Chris,
>> > >
>> > >Thanks for your emails from Friday and Saturday. They were extremely
>> > >helpful and I have now worked my way through all of my setup issues
>> > >so far. I was using the new style CASE-PGE keys because Rishi used
>> > >those in his working example and at one point I had been trying to
>> > >mimic that example. I didn't know anything about the legacy mode
>> > >flag. But, after changing the keys to the old style, everything
>> > >started to come together a little faster.
>> > >
>> > >To start my crawler at the beginning of my pipeline, I chose the
>> > >approach that Rishi had taken where the pgeConfig starts the
>> > >crawler_launcher script directly rather than rely on the definition
>> > >of the output files to trigger the crawler to run. This was because
>> > >the met extractor I am using is a perl script, and your DRAT system
>> > >only shows how to specify a class for the metFileWriterClass
>> > >attribute. I could not find anything to show me how to specify a perl
>> > >script as the met file
>> > writer instead.
>> > >Also, having to specify the file types to crawl for by using a
>> > >regular expression in the config file seemed like an unnecessary
>> > >duplication of the definition of the files types to crawl for (since
>> > >the regular expression just specifies a mime-type that is already
>> > >defined in mime-types.xml). I didn't want to specify the mime-type in
>> > >two different places.
>> > >
>> > >So, now I've been through the process of setting up my CAS-PGE to
>> > >ingest the raw science and spacecraft housekeeping telemetry files,
>> > >and run a postIngestSuccess action to copy some of those files to a
>> > >new directory for further processing.
>> > >
>> > >The next step in my pipeline may be a little tricky. I need to query
>> > >the filemgr and pass the results of the query (it could be a long
>> > >list) as an input parameter to the first algorithm in the pipeline
>> > >which, happens to be another perl script. Does anyone know how I
>> > >might do this in a PGE config file?
>> > >
>> > >Thanks!!
>> > >Val
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >Valerie A. Mallder
>> > >New Horizons Deputy Mission System Engineer Johns Hopkins
>> > >University/Applied Physics Laboratory
>> > >
>> > >
>> > >> -----Original Message-----
>> > >> From: Mattmann, Chris A (3980)
>> > >> [mailto:chris.a.mattmann@jpl.nasa.gov]
>> > >> Sent: Friday, October 10, 2014 7:59 PM
>> > >> To: dev@oodt.apache.org
>> > >> Subject: Re: Failed to build PgeConfig, exception in
>> > >> PathUtils.doDynamicReplacement
>> > >>
>> > >> Thanks Val you are close!
>> > >>
>> > >> Looking at what you show below, I think the issue is one of the
>> > >>following:
>> > >>
>> > >> 1. You are using the new style CAS-PGE Keys. Try using the old ones
>> > >>(e.g., the ones present here:
>> > >>
>> > >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/ex
>> > >>am
>> > >>ple
>> > >>s/
>> > >> WorkflowTask/tasks.xml
>> > >>
>> > >> (note the ?_? and not the ?/? used in keys).
>> > >> These keys are used by CAS-PGE if you see something in your wmgr
>> > >>bin script (or your resource manager batch_stub script) stating
>> > >>?legacyMode=true?).
>> > >> If you
>> > >> are using RADIX, I believe that?s the case:
>> > >>
>> > >> http://s.apache.org/hO
>> > >>
>> > >> 2. Regarding better building environments. The m2e plugin for
>> > >>Eclipse is fantastic nowadays and can literally checkout a
>> > >>multi-module Maven project from SVN (when paired with Subversive as
>> > >>a plugin or Subclipse). Another thing to check out is this page on
>>the wiki:
>> > >>
>> > >>
>> > >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develo
>> > >>pe
>> > >>r+H
>> > >>el
>> > >> p
>> > >>
>> > >>
>> > >> Let me know if that fixes it. Sorry for all the trouble :) Trust me
>> > >>it will be worth it.
>> > >> CAS-PGE is awesome once working.
>> > >>
>> > >> Cheers,
>> > >> Chris
>> > >>
>> > >>
>> > >>
>> >
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> +++++
>> > >> Chris Mattmann, Ph.D.
>> > >> Chief Architect
>> > >> Instrument Software and Science Data Systems Section (398) NASA Jet
>> > >> Propulsion Laboratory Pasadena, CA 91109 USA
>> > >> Office: 168-519, Mailstop: 168-527
>> > >> Email: chris.a.mattmann@nasa.gov
>> > >> WWW: http://sunset.usc.edu/~mattmann/
>> > >>
>> >
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> +++++
>> > >> Adjunct Associate Professor, Computer Science Department University
>> > >> of Southern California, Los Angeles, CA 90089 USA
>> > >>
>> >
>> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> > >> +++++
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: <Mallder>, Valerie <Va...@jhuapl.edu>
>> > >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > >> Date: Friday, October 10, 2014 at 3:40 PM
>> > >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
>> > >> Subject: Failed to build PgeConfig, exception in
>> > >>PathUtils.doDynamicReplacement
>> > >>
>> > >> >Hi All,
>> > >> >
>> > >> >I'm still working on this! Still trying to get a CA-PGE Task to
>>run.
>> > >> >I'm almost there. Now it at least trys to build the PGE config
>>file.
>> > >>But the
>> > >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking
>>up a
>> > >> >null string somewhere, but it also looks like it is crashing in a
>> > >> >strange place. It is unable to read the .xml file that contains
>> > >> >my <pgeConfig>...</pgeConfig> but, the correct path and file are
>> > >> >shown in the output log messages I've include below which also
>> > >> >shows the stack trace. I looks like it is failing during a
>> > >> >recursive call to PathUtils.doDynamicReplacement. I saw some
>> > >> >chatter about some prior errors in this code in the mailing list
>>archives from
>> 2010 and 2012.
>> > >> >So I am hoping someone might remember and be able to tell me if I
>> > >> >am doing something that is causing this error. I included my
>> > >> >tasks.xml and fei-crawler-pge-config.xml file after the log
>>messages.
>> > >> >
>> > >> >At this point, I think need to start running these processes in an
>> > >> >environment where I can debug this better. The runtime output and
>> > >> >stack traces simply aren't enough for me to track this down. I am
>> > >> >newbie to using eclipse and mvn, so, does anyone have some notes
>> > >> >on the best way to import oodt-0.7 into an exclipe workspace and
>> > >> >tell it to build it using mvn?? I am hoping I can leverage other
>> > >> >peoples knowledge of how to do this so that I can do it quickly
>> > >> >and not waste another week. I imported the oodt-0.7 directory into
>> > >> >eclipse so I could view the files easier, but I just made eclipse
>> > >> >link to the where the files are located rather that copy them into
>> > >> >a workspace folder. If there's a better way, please tell me. And
>> > >> >now, how do I tell exclipse
>> > >>to build
>> > >> stuff?
>> > >> >
>> > >> >Thanks!!
>> > >> >Valerie
>> > >> >
>> > >> >
>> > >> >
>> > >> >Using CATALINA_BASE:
>> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca
>> > >> >t
>> > >> >Using CATALINA_HOME:
>> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca
>> > >> >t
>> > >> >Using CATALINA_TMPDIR:
>> > >>
>> > >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
>> > >>>/t
>> > >>>emp
>> > >> >Using JRE_HOME:
>>/project/jedi/users/jedi-pipeline/jdk1.7.0_55
>> > >> >Workflow Manager started PID file
>> > >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work
>> > >> >fl
>> > >> >ow/
>> > >> >run
>> > >> >/cas.workflow.pid).
>> > >> >Oct 10, 2014 5:53:52 PM
>> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
>> > >> >loadProperties
>> > >> >INFO: Loading Workflow Manager Configuration Properties from:
>> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work
>> > >> >fl
>> > >> >ow/
>> > >> >etc
>> > >> >/workflow.properties]
>> > >> >Oct 10, 2014 5:53:52 PM
>> > >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactor
>> > >> >y
>> > >> >getResmgrUrl
>> > >> >INFO: No Resource Manager URL provided or malformed URL: executing
>> > >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM
>> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
>> > >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM
>> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
>> > >> >handleEvent
>> > >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10,
>> > >> >2014
>> > >> >5:54:12 PM
>> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
>> > >> >handleEvent
>> > >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for
>> > >> >event startJediPipeline Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >checkTaskRequiredMetadata
>> > >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields
>> > >> >Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >executeTaskLocally
>> > >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014
>> > >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >INFO: Converting workflow configuration to static metadata...
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl]
>> > >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key =
>> > >> >[PGETask/Ingest/CrawlerConfigFile]
>> > >> >value =
>> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
>> > >> >/c
>> > >> >raw
>> > >> >ler
>> > >> >/policy/crawler-config.xml]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGETask/Name] value =
>> > >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds]
>> > >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value
>> > >> >= [true] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key =
>> > >> >[PGETask/Query/ClientTransferServiceFactory] value =
>> > >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
>> > >> >] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGE_HOME] value =
>> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key =
>> > >> >[PGETask/Query/FileManagerUrl] value = [http://localhost:9000] Oct
>> > >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key =
>> > >> >[PGETask/Ingest/MimeExtractorRepo]
>> > >> >value =
>> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
>> > >> >/e
>> > >> >xte
>> > >> >nsi
>> > >> >ons/policy/mime-extractor-map.xml]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath]
>> > >> >value =
>> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
>> > >> >/e
>> > >> >xte
>> > >> >nsi
>> > >> >ons/config/fei-crawler-pge-config.xml]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding static metadata: key =
>> > >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >INFO: Loading workflow context metadata...
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
>> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding dynamic metadata: key = [JobId] value =
>> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value
>> > >> >= [http://slothrop.jhuapl.edu:9001] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding dynamic metadata: key = [TaskId] value =
>> > >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeMetadata
>> > >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
>> > >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeConfig
>> > >> >INFO: Create PgeConfig...
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >createPgeConfig
>> > >> >INFO: Using default PgeConfigBuilder:
>> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
>> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
>> > >> >run
>> > >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
>> > >> >parse
>> > >> >value: null
>> > >> >java.io.IOException: Failed to build PgeConfig : Failed to parse
>>value:
>> > >> >null
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
>> > >> >le
>> > >> >Pge
>> > >> >Con
>> > >> >figBuilder.java:87)
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns
>> > >> >ta
>> > >> >nce
>> > >> >.ja
>> > >> >va:230)
>> > >> > at
>> > >>
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>> > >> > at
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >.ex
>> > >> >ecu
>> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
>> > >> > at
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >.ru
>> > >> >n(I
>> > >> >terativeWorkflowProcessorThread.java:321)
>> > >> > at
>> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> > >>Source)
>> > >> > at java.lang.Thread.run(Thread.java:745)
>> > >> >Caused by: java.lang.Exception: Failed to parse value: null
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
>> > >> >le
>> > >> >Pge
>> > >> >Con
>> > >> >figBuilder.java:77)
>> > >> > ... 6 more
>> > >> >Caused by: java.lang.NullPointerException
>> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>> > >> > at java.util.regex.Matcher.reset(Matcher.java:308)
>> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
>> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
>> > >> > at
>> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR
>> > >> >ep
>> > >> >lac
>> > >> >eme
>> > >> >nt(PathUtils.java:321)
>> > >> > at
>> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P
>> > >> >at
>> > >> >hUt
>> > >> >ils
>> > >> >.java:96)
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
>> > >> > ... 8 more
>> > >> >
>> > >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstan
>> > >> >ce
>> > >> >Exc
>> > >> >ept
>> > >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
>> > >> >parse
>> > >> >value: null
>> > >> > at
>> > >>
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
>> > >> > at
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >.ex
>> > >> >ecu
>> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
>> > >> > at
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >.ru
>> > >> >n(I
>> > >> >terativeWorkflowProcessorThread.java:321)
>> > >> > at
>> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> > >>Source)
>> > >> > at java.lang.Thread.run(Thread.java:745)
>> > >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed
>> > >> >to parse value: null
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
>> > >> >le
>> > >> >Pge
>> > >> >Con
>> > >> >figBuilder.java:87)
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns
>> > >> >ta
>> > >> >nce
>> > >> >.ja
>> > >> >va:230)
>> > >> > at
>> > >>
>>>org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
>> > >> > ... 4 more
>> > >> >Caused by: java.lang.Exception: Failed to parse value: null
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
>> > >> > at
>> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
>> > >> >le
>> > >> >Pge
>> > >> >Con
>> > >> >figBuilder.java:77)
>> > >> > ... 6 more
>> > >> >Caused by: java.lang.NullPointerException
>> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
>> > >> > at java.util.regex.Matcher.reset(Matcher.java:308)
>> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
>> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
>> > >> > at
>> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR
>> > >> >ep
>> > >> >lac
>> > >> >eme
>> > >> >nt(PathUtils.java:321)
>> > >> > at
>> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P
>> > >> >at
>> > >> >hUt
>> > >> >ils
>> > >> >.java:96)
>> > >> > at
>> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
>> > >> > ... 8 more
>> > >> >Oct 10, 2014 5:54:13 PM
>> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
>> > >> >ad
>> > >> >executeTaskLocally
>> > >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally:
>> > >>Message:
>> > >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse
>>value:
>> > >> >null
>> > >> >
>> > >> >
>> > >> >
>> > >> >
>> > >> >Content of tasks.xml
>> > >> >
>> > >> >
>> > >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
>> > >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
>> > >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance">
>> > >> > <conditions/>
>> > >> >
>> > >> > <configuration>
>> > >> > <property name="PGETask/Name"
>>value="feiCrawlerTaskName"/>
>> > >> > <property name="PGETask/ConfigFilePath"
>> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> > >> >de plo y/e xtensions/config/fei-crawler-pge-config.xml"/>
>> > >> > <property name="PGETask/DumpMetadata" value="true"/>
>> > >> > <property name="PGETask/WorkflowManagerUrl"
>> > >> >value="http://localhost:9001" />
>> > >> > <property name="PGETask/Query/FileManagerUrl"
>> > >> >value="http://localhost:9000" />
>> > >> > <property name="PGETask/Ingest/FileManagerUrl"
>> > >> >value="http://localhost:9000"/>
>> > >> >
>> > >> > <property
>>name="PGETask/Query/ClientTransferServiceFactory"
>> > >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferF
>> > >> >ac
>> > >> >tor
>> > >> >y"/
>> > >> >>
>> > >> > <property name="PGETask/Ingest/CrawlerConfigFile"
>> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> > >> >de
>> > >> >plo
>> > >> >y/c
>> > >> >rawler/policy/crawler-config.xml"/>
>> > >> > <property name="PGETask/Ingest/MimeExtractorRepo"
>> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
>> > >> >de plo y/e xtensions/policy/mime-extractor-map.xml"/>
>> > >> > <property name="PGETask/Ingest/ActionIds"
>> > >> >value="MoveFileToLevel0Dir"/>
>> > >> > <property name="PGE_HOME"
>> > >>
>> >
>>>>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pg
>>>>>e"
>> > >>>/>
>> > >> > </configuration>
>> > >> >
>> > >> > <requiredMetFields/>
>> > >> > </task>
>> > >> ></cas:tasks>
>> > >> >
>> > >> >
>> > >> >Contents of fei-crawler-pge-config.xml
>> > >> >
>> > >> >
>> > >> ><pgeConfig>
>> > >> >
>> > >> > <!-- How to run the PGE -->
>> > >> > <!-just echoing the current directory to a file so I can see if
>> > >> >this thing ever starts -->
>> > >> > <exe
>> > >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/
>> > >> >da
>> > >> >ta/
>> > >> >pge
>> > >> >/jobs" shell="/bin/sh">
>> > >> > <cmd>echo "Current Working Directory is `pwd`" >
>> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/
>> > >> >pg
>> > >> >e.l
>> > >> >og<
>> > >> >/cmd>
>> > >> > </exe>
>> > >> >
>> > >> > <!-- Files to ingest -->
>> > >> > <output>
>> > >> > <!-- trying this approach. Telling PGE there is output files
>> > >> >should invoke the crawler. -->
>> > >> >
>> > >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
>> > >> >/d
>> > >> >ata
>> > >> >/st
>> > >> >aging"/>
>> > >> > </output>
>> > >> >
>> > >> > <!-- Custom metadata to add to output files -->
>> > >> > <customMetadata>
>> > >> >
>> > >> > <!-- each of these directories exist -->
>> > >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
>> > >> >envReplace="true"/>
>> > >> > <metadata key="JobInputDir" val="[JobDir]/input"/>
>> > >> > <metadata key="JobOutputDir" val="[JobDir]/output"/>
>> > >> > <metadata key="JobLogDir" val="[JobDir]/logs"/>
>> > >> >
>> > >> > </customMetadata>
>> > >> ></pgeConfig>
>> > >> >
>> > >> >Valerie A. Mallder
>> > >> >
>> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins
>> > >> >University/Applied Physics Laboratory
>> > >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
>> > >> >240-228-7846 (Office) 410-504-2233 (Blackberry)
>> > >> >
>> > >
>> >
>
RE: SQL parsing error in PgeConfig
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
Found it! Ugh! I'm an idiot! Got the sql to work. Now moving on to figuring out why the class in my jar file can't be found. But that's progress! Yay!
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Mallder, Valerie [mailto:Valerie.Mallder@jhuapl.edu]
> Sent: Wednesday, October 15, 2014 5:33 PM
> To: dev@oodt.apache.org
> Subject: SQL parsing error in PgeConfig
>
> Hi Chris,
>
> I played with the SQL statement in my pge config file today and keep getting a
> parsing error. I tooked at your PgeConfig_RatAggregator.xml (oh, you have a cut-
> and-paste error on line 31, '[Ampersand]' should be '[Exclamation]' if I understand
> correctly what you are doing there. And this typo propagated its way to the other
> xml files as well), and I have tried to mimic the parts that apply to what I am trying
> to do and I'm stumped by the SQL parsing error.
>
> I am doing a very simple query just to try to prove that I *can* do a query. I didn't
> really need the FORMAT agreement, but I wasn't sure the right syntax for leaving it
> out. So I just specified the metadata key I'm searching for in the FORMAT
> arguement. So here's what I have.
>
> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT FileLocation
> FROM EngineeringFile }"/>
>
> One difference between this statement and yours is that you used $ signs in your
> FORMAT specification. I have tried it both ways, with dollar signs and without
> dollar signs and still get the parse error. (What is the significance of the "$"?)
>
> Here's the output from oodt.out, and my pge config file and tasks.xml follow
> afterwards. The stack trace looks familiar from the standpoint that I've seen parse
> errors before and fixed them. But, this one points out a suspicious looking
> exception for a malformed URL for some reason. However, the output from the log
> file indicates all of my urls are fine. No pge metadata is written for this task and no
> script is created. Please let me know if you see anything obvious that I'm missing.
>
> Thanks,
> Val
>
>
>
> Oct 15, 2014 5:15:02 PM
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> checkTaskRequiredMetadata
> INFO: Task: [DB-Builder-Task] has no required metadata fields Oct 15, 2014
> 5:15:03 PM
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> executeTaskLocally
> INFO: Executing task: [DB-Builder-Task] locally Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> INFO: Converting workflow configuration to static metadata...
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Ingest_CrawlerConfigFile] value
> = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/crawler/policy/crawler-config.xml]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PCS_WorkflowManagerUrl] value =
> [http://localhost:9001] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Ingest_FileManagerUrl] value =
> [http://localhost:9000] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding static metadata: key = [FEI_DROP_DIR] value =
> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/data/staging]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGE_HOME] value =
> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key =
> [PGETask_Query_ClientTransferServiceFactory] value =
> [org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Ingest_ActionIds] value =
> [MoveFileToLevel0Dir] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Ingest_MimeExtractorRepo]
> value = [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/extensions/policy/mime-extractor-map.xml]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_DumpMetadata] value = [true]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Query_FileManagerUrl] value =
> [http://localhost:9000] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_Name] value = [DB-Builder-
> Task] Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding static metadata: key = [PGETask_ConfigFilePath] value =
> [/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/extensions/config/jedi-conduit-db-builder-config.xml]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> INFO: Loading workflow context metadata...
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding dynamic metadata: key = [WorkflowInstId] value = [44f791c9-
> 54b0-11e4-a032-c9e22d0219d9]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding dynamic metadata: key = [JobId] value = [44f791c9-54b0-11e4-
> a032-c9e22d0219d9]
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeMetadata
> FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value =
> [http://slothrop.jhuapl.edu:9001] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding dynamic metadata: key = [TaskId] value =
> [urn:oodt:dbBuilderTask] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeMetadata
> FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
> [slothrop.jhuapl.edu] Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.pge.PGETaskInstance createPgeConfig
> INFO: Create PgeConfig...
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance
> createPgeConfig
> INFO: Using default PgeConfigBuilder:
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
> Oct 15, 2014 5:15:03 PM org.apache.oodt.cas.pge.PGETaskInstance run
> SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig
> Builder.java:87)
> at
> org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja
> va:230)
> at
> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> at
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeT
> askLocally(IterativeWorkflowProcessorThread.java:574)
> at
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(Iterati
> veWorkflowProcessorThread.java:321)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.Exception: Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> at
> org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(X
> mlFilePgeConfigBuilder.java:165)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFile
> PgeConfigBuilder.java:120)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig
> Builder.java:83)
> ... 6 more
> Caused by: java.net.MalformedURLException
> at java.net.URL.<init>(URL.java:619)
> at java.net.URL.<init>(URL.java:482)
> at java.net.URL.<init>(URL.java:431)
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
> ... 11 more
> Caused by: java.lang.NullPointerException
> at java.net.URL.<init>(URL.java:524)
> ... 14 more
>
> org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstanceException
> : PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> at
> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
> at
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.executeT
> askLocally(IterativeWorkflowProcessorThread.java:574)
> at
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread.run(Iterati
> veWorkflowProcessorThread.java:321)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig
> Builder.java:87)
> at
> org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.ja
> va:230)
> at
> org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> ... 4 more
> Caused by: java.lang.Exception: Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> at
> org.apache.oodt.cas.pge.util.XmlHelper.getMetadataValues(XmlHelper.java:245)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadCustomMetadata(X
> mlFilePgeConfigBuilder.java:165)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.loadConfigFile(XmlFile
> PgeConfigBuilder.java:120)
> at
> org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfig
> Builder.java:83)
> ... 6 more
> Caused by: java.net.MalformedURLException
> at java.net.URL.<init>(URL.java:619)
> at java.net.URL.<init>(URL.java:482)
> at java.net.URL.<init>(URL.java:431)
> at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:492)
> ... 11 more
> Caused by: java.lang.NullPointerException
> at java.net.URL.<init>(URL.java:524)
> ... 14 more
> Oct 15, 2014 5:15:03 PM
> org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThread
> executeTaskLocally
> WARNING: Exception executing task: [DB-Builder-Task] locally: Message:
> PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> SQL(FORMAT='FileLocation') { SELECT FileLocation FROM EngineeringFile }
>
> <!--
> ** Filename: jedi-conduit-db-builder-config.xml
> -->
> <pgeConfig>
>
> <import file="pge-common-config.xml" />
>
> <!-- Note: do not use the "shell" attribute, "shellType is the correct attribute here"-
> ->
> <exe dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/data/pge/jobs" shellType="/usr/local/bin/bash">
> <cmd>echo "Current Working Directory is `pwd`" > [JobLogDir]/db-
> builder.log</cmd>
> <cmd>echo "Current Environment is `/bin/env`" >> [JobLogDir]/db-
> builder.log</cmd>
>
> <cmd>[JavaHome] -cp [JediConduitJar] [LoadClass] [DirList]
> [Arguements]</cmd>
> </exe>
>
> <!-- Custom metadata to add to output files -->
> <customMetadata>
> <metadata key="LoadClass" val="conduit.jedi.utils.JEDIFastDBBuilder" />
> <metadata key="TelemetryFileDb" val="[JobDir]/valsTelemetryFile.db" />
> <metadata key="Arguements" val="[TelemetryFileDb]" />
>
> <metadata key="DirList" val="SQL(FORMAT='FileLocation') { SELECT
> FileLocation FROM EngineeringFile }" />
>
> <!-- SQL example statement from the code
> SQL
> (FORMAT='FileLocation/Filename',SORT_BY='FileSize',FILTER=<StartDateTime,En
> dDateTime,DataVersion,TakeHighestPriority) { SELECT
> FileLocation,Filename,FileSize FROM IASI_L1C WHERE ProductionDateTime >=
> '2007-12-01T00:00:00.000000Z' }
> -->
> </customMetadata>
> </pgeConfig>
>
>
> <!--
> ** Filename: tasks.xml
> -->
>
> <task id="urn:oodt:dbBuilderTask" name="DB-Builder-Task"
> class="org.apache.oodt.cas.pge.StdPGETaskInstance">
> <conditions/>
>
> <configuration>
> <property name="PGETask_Name" value="DB-Builder-Task"/>
> <property name="PGETask_ConfigFilePath"
> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/extensions/config/jedi-conduit-db-builder-config.xml" envReplace="true" />
> <property name="PGETask_DumpMetadata" value="true"/>
> <property name="PCS_WorkflowManagerUrl" value="http://localhost:9001"
> />
> <property name="PGETask_Query_FileManagerUrl"
> value="http://localhost:9000" />
> <property name="PGETask_Ingest_FileManagerUrl"
> value="http://localhost:9000"/>
>
> <property name="PGETask_Query_ClientTransferServiceFactory"
> value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory"/>
> <property name="PGETask_Ingest_CrawlerConfigFile"
> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/crawler/policy/crawler-config.xml"/>
> <property name="PGETask_Ingest_MimeExtractorRepo"
> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/extensions/policy/mime-extractor-map.xml"/>
> <property name="PGETask_Ingest_ActionIds"
> value="MoveFileToLevel0Dir"/>
> <property name="PGE_HOME"
> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"/>
>
> <property name="FEI_DROP_DIR"
> value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> deploy/data/staging"/>
> </configuration>
> <requiredMetFields/>
> </task>
>
>
> Valerie A. Mallder
> New Horizons Deputy Mission System Engineer Johns Hopkins University/Applied
> Physics Laboratory
>
> > -----Original Message-----
> > From: Chris Mattmann [mailto:chris.mattmann@gmail.com]
> > Sent: Tuesday, October 14, 2014 4:11 PM
> > To: dev@oodt.apache.org
> > Subject: Re: Failed to build PgeConfig, exception in
> > PathUtils.doDynamicReplacement
> >
> > Awesome Val, glad it helped and that you are moving further!
> >
> > Here?s a quick example of how to call your perl script as an extractor
> > in CAS-PGE (barring yes, the duplication of specifying a regExp to
> > select the type to run the extractor on, but I?ll get to that later):
> >
> > Under <output>:
> >
> > <files regExp=?\\*.hdf"
> >
> > metFileWriterClass="org.apache.oodt.cas.pge.writers.ExternalExtractorM
> > etWri
> > ter"
> > args=?/path/to/your-extern-config.xml?>
> >
> >
> > Your extern-config.xml file should work like this:
> >
> > http://svn.apache.org/repos/asf/oodt/trunk/metadata/src/main/resources
> > /exam
> > ples/extern-config.xml
> >
> >
> > Note that the ExternExtractor is documented here:
> >
> > http://oodt.apache.org/components/maven/metadata/user/basic.html
> >
> >
> > That will get you your Perl script integrated as an extractor into the system.
> >
> > On to your question about selecting information from the File Manager:
> >
> > Look up the SQL(.. portions of the DRAT PGE configs to get an idea of
> > how to do this. In short, yes you can do it, and the way to do it is
> > to use the special CAS- PGE SQL-like language that you can prefix your
> > met key with (make sure envReplace is turned on). You can do like
> > SQL([options]){ SELECT FROM <product type name> WHERE <criteria>
> >
> > The results of your above query are stored into the metadata field
> > name in your CAS-PGE config file.
> >
> > I?ll walk you through more of this later, but have to run to a meeting.
> >
> > Gluck!
> >
> > Cheers,
> > Chris
> >
> >
> > ------------------------
> > Chris Mattmann
> > chris.mattmann@gmail.com
> >
> >
> >
> >
> > -----Original Message-----
> > From: "Mallder, Valerie" <Va...@jhuapl.edu>
> > Reply-To: <de...@oodt.apache.org>
> > Date: Tuesday, October 14, 2014 at 12:58 PM
> > To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > Subject: RE: Failed to build PgeConfig, exception in
> > PathUtils.doDynamicReplacement
> >
> > >Hi Chris,
> > >
> > >Thanks for your emails from Friday and Saturday. They were extremely
> > >helpful and I have now worked my way through all of my setup issues
> > >so far. I was using the new style CASE-PGE keys because Rishi used
> > >those in his working example and at one point I had been trying to
> > >mimic that example. I didn't know anything about the legacy mode
> > >flag. But, after changing the keys to the old style, everything
> > >started to come together a little faster.
> > >
> > >To start my crawler at the beginning of my pipeline, I chose the
> > >approach that Rishi had taken where the pgeConfig starts the
> > >crawler_launcher script directly rather than rely on the definition
> > >of the output files to trigger the crawler to run. This was because
> > >the met extractor I am using is a perl script, and your DRAT system
> > >only shows how to specify a class for the metFileWriterClass
> > >attribute. I could not find anything to show me how to specify a perl
> > >script as the met file
> > writer instead.
> > >Also, having to specify the file types to crawl for by using a
> > >regular expression in the config file seemed like an unnecessary
> > >duplication of the definition of the files types to crawl for (since
> > >the regular expression just specifies a mime-type that is already
> > >defined in mime-types.xml). I didn't want to specify the mime-type in
> > >two different places.
> > >
> > >So, now I've been through the process of setting up my CAS-PGE to
> > >ingest the raw science and spacecraft housekeeping telemetry files,
> > >and run a postIngestSuccess action to copy some of those files to a
> > >new directory for further processing.
> > >
> > >The next step in my pipeline may be a little tricky. I need to query
> > >the filemgr and pass the results of the query (it could be a long
> > >list) as an input parameter to the first algorithm in the pipeline
> > >which, happens to be another perl script. Does anyone know how I
> > >might do this in a PGE config file?
> > >
> > >Thanks!!
> > >Val
> > >
> > >
> > >
> > >
> > >
> > >Valerie A. Mallder
> > >New Horizons Deputy Mission System Engineer Johns Hopkins
> > >University/Applied Physics Laboratory
> > >
> > >
> > >> -----Original Message-----
> > >> From: Mattmann, Chris A (3980)
> > >> [mailto:chris.a.mattmann@jpl.nasa.gov]
> > >> Sent: Friday, October 10, 2014 7:59 PM
> > >> To: dev@oodt.apache.org
> > >> Subject: Re: Failed to build PgeConfig, exception in
> > >> PathUtils.doDynamicReplacement
> > >>
> > >> Thanks Val you are close!
> > >>
> > >> Looking at what you show below, I think the issue is one of the
> > >>following:
> > >>
> > >> 1. You are using the new style CAS-PGE Keys. Try using the old ones
> > >>(e.g., the ones present here:
> > >>
> > >>http://svn.apache.org/repos/asf/oodt/trunk/pge/src/main/resources/ex
> > >>am
> > >>ple
> > >>s/
> > >> WorkflowTask/tasks.xml
> > >>
> > >> (note the ?_? and not the ?/? used in keys).
> > >> These keys are used by CAS-PGE if you see something in your wmgr
> > >>bin script (or your resource manager batch_stub script) stating
> > >>?legacyMode=true?).
> > >> If you
> > >> are using RADIX, I believe that?s the case:
> > >>
> > >> http://s.apache.org/hO
> > >>
> > >> 2. Regarding better building environments. The m2e plugin for
> > >>Eclipse is fantastic nowadays and can literally checkout a
> > >>multi-module Maven project from SVN (when paired with Subversive as
> > >>a plugin or Subclipse). Another thing to check out is this page on the wiki:
> > >>
> > >>
> > >>https://cwiki.apache.org/confluence/display/OODT/OODT+Eclipse+Develo
> > >>pe
> > >>r+H
> > >>el
> > >> p
> > >>
> > >>
> > >> Let me know if that fixes it. Sorry for all the trouble :) Trust me
> > >>it will be worth it.
> > >> CAS-PGE is awesome once working.
> > >>
> > >> Cheers,
> > >> Chris
> > >>
> > >>
> > >>
> >
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> +++++
> > >> Chris Mattmann, Ph.D.
> > >> Chief Architect
> > >> Instrument Software and Science Data Systems Section (398) NASA Jet
> > >> Propulsion Laboratory Pasadena, CA 91109 USA
> > >> Office: 168-519, Mailstop: 168-527
> > >> Email: chris.a.mattmann@nasa.gov
> > >> WWW: http://sunset.usc.edu/~mattmann/
> > >>
> >
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> +++++
> > >> Adjunct Associate Professor, Computer Science Department University
> > >> of Southern California, Los Angeles, CA 90089 USA
> > >>
> >
> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > >> +++++
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: <Mallder>, Valerie <Va...@jhuapl.edu>
> > >> Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > >> Date: Friday, October 10, 2014 at 3:40 PM
> > >> To: "dev@oodt.apache.org" <de...@oodt.apache.org>
> > >> Subject: Failed to build PgeConfig, exception in
> > >>PathUtils.doDynamicReplacement
> > >>
> > >> >Hi All,
> > >> >
> > >> >I'm still working on this! Still trying to get a CA-PGE Task to run.
> > >> >I'm almost there. Now it at least trys to build the PGE config file.
> > >>But the
> > >> >XmlFilePgeConfigBuilder is failing. I looks like it is picking up a
> > >> >null string somewhere, but it also looks like it is crashing in a
> > >> >strange place. It is unable to read the .xml file that contains
> > >> >my <pgeConfig>...</pgeConfig> but, the correct path and file are
> > >> >shown in the output log messages I've include below which also
> > >> >shows the stack trace. I looks like it is failing during a
> > >> >recursive call to PathUtils.doDynamicReplacement. I saw some
> > >> >chatter about some prior errors in this code in the mailing list archives from
> 2010 and 2012.
> > >> >So I am hoping someone might remember and be able to tell me if I
> > >> >am doing something that is causing this error. I included my
> > >> >tasks.xml and fei-crawler-pge-config.xml file after the log messages.
> > >> >
> > >> >At this point, I think need to start running these processes in an
> > >> >environment where I can debug this better. The runtime output and
> > >> >stack traces simply aren't enough for me to track this down. I am
> > >> >newbie to using eclipse and mvn, so, does anyone have some notes
> > >> >on the best way to import oodt-0.7 into an exclipe workspace and
> > >> >tell it to build it using mvn?? I am hoping I can leverage other
> > >> >peoples knowledge of how to do this so that I can do it quickly
> > >> >and not waste another week. I imported the oodt-0.7 directory into
> > >> >eclipse so I could view the files easier, but I just made eclipse
> > >> >link to the where the files are located rather that copy them into
> > >> >a workspace folder. If there's a better way, please tell me. And
> > >> >now, how do I tell exclipse
> > >>to build
> > >> stuff?
> > >> >
> > >> >Thanks!!
> > >> >Valerie
> > >> >
> > >> >
> > >> >
> > >> >Using CATALINA_BASE:
> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca
> > >> >t
> > >> >Using CATALINA_HOME:
> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomca
> > >> >t
> > >> >Using CATALINA_TMPDIR:
> > >>
> > >>>/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/tomcat
> > >>>/t
> > >>>emp
> > >> >Using JRE_HOME: /project/jedi/users/jedi-pipeline/jdk1.7.0_55
> > >> >Workflow Manager started PID file
> > >> >(/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work
> > >> >fl
> > >> >ow/
> > >> >run
> > >> >/cas.workflow.pid).
> > >> >Oct 10, 2014 5:53:52 PM
> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> > >> >loadProperties
> > >> >INFO: Loading Workflow Manager Configuration Properties from:
> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/work
> > >> >fl
> > >> >ow/
> > >> >etc
> > >> >/workflow.properties]
> > >> >Oct 10, 2014 5:53:52 PM
> > >> >org.apache.oodt.cas.workflow.engine.ThreadPoolWorkflowEngineFactor
> > >> >y
> > >> >getResmgrUrl
> > >> >INFO: No Resource Manager URL provided or malformed URL: executing
> > >> >jobs locally. URL: [null] Oct 10, 2014 5:53:52 PM
> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager <init>
> > >> >INFO: Workflow Manager started by malldva1 Oct 10, 2014 5:54:12 PM
> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> > >> >handleEvent
> > >> >INFO: WorkflowManager: Received event: startJediPipeline Oct 10,
> > >> >2014
> > >> >5:54:12 PM
> > >> >org.apache.oodt.cas.workflow.system.XmlRpcWorkflowManager
> > >> >handleEvent
> > >> >INFO: WorkflowManager: Workflow jediWorkflowName retrieved for
> > >> >event startJediPipeline Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >checkTaskRequiredMetadata
> > >> >INFO: Task: [feiCrawlerTaskName] has no required metadata fields
> > >> >Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >executeTaskLocally
> > >> >INFO: Executing task: [feiCrawlerTaskName] locally Oct 10, 2014
> > >> >5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >INFO: Converting workflow configuration to static metadata...
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGETask/WorkflowManagerUrl]
> > >> >value = [http://localhost:9001] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key =
> > >> >[PGETask/Ingest/CrawlerConfigFile]
> > >> >value =
> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
> > >> >/c
> > >> >raw
> > >> >ler
> > >> >/policy/crawler-config.xml]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGETask/Name] value =
> > >> >[feiCrawlerTaskName] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGETask/Ingest/ActionIds]
> > >> >value = [MoveFileToLevel0Dir] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGETask/DumpMetadata] value
> > >> >= [true] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key =
> > >> >[PGETask/Query/ClientTransferServiceFactory] value =
> > >> >[org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferFactory
> > >> >] Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGE_HOME] value =
> > >> >[/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key =
> > >> >[PGETask/Query/FileManagerUrl] value = [http://localhost:9000] Oct
> > >> >10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key =
> > >> >[PGETask/Ingest/MimeExtractorRepo]
> > >> >value =
> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
> > >> >/e
> > >> >xte
> > >> >nsi
> > >> >ons/policy/mime-extractor-map.xml]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key = [PGETask/ConfigFilePath]
> > >> >value =
> > >> >[file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
> > >> >/e
> > >> >xte
> > >> >nsi
> > >> >ons/config/fei-crawler-pge-config.xml]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding static metadata: key =
> > >> >[PGETask/Ingest/FileManagerUrl] value = [http://localhost:9000]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >INFO: Loading workflow context metadata...
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding dynamic metadata: key = [WorkflowInstId] value =
> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding dynamic metadata: key = [JobId] value =
> > >> >[f8730997-50c7-11e4-b9aa-57625eee7ebd]
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding dynamic metadata: key = [WorkflowManagerUrl] value
> > >> >= [http://slothrop.jhuapl.edu:9001] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding dynamic metadata: key = [TaskId] value =
> > >> >[urn:oodt:feiCrawlerTaskId] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeMetadata
> > >> >FINEST: Adding dynamic metadata: key = [ProcessingNode] value =
> > >> >[slothrop.jhuapl.edu] Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeConfig
> > >> >INFO: Create PgeConfig...
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >createPgeConfig
> > >> >INFO: Using default PgeConfigBuilder:
> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder
> > >> >Oct 10, 2014 5:54:13 PM org.apache.oodt.cas.pge.PGETaskInstance
> > >> >run
> > >> >SEVERE: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
> > >> >parse
> > >> >value: null
> > >> >java.io.IOException: Failed to build PgeConfig : Failed to parse value:
> > >> >null
> > >> > at
> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
> > >> >le
> > >> >Pge
> > >> >Con
> > >> >figBuilder.java:87)
> > >> > at
> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns
> > >> >ta
> > >> >nce
> > >> >.ja
> > >> >va:230)
> > >> > at
> > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> > >> > at
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >.ex
> > >> >ecu
> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> > >> > at
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >.ru
> > >> >n(I
> > >> >terativeWorkflowProcessorThread.java:321)
> > >> > at
> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> > >>Source)
> > >> > at java.lang.Thread.run(Thread.java:745)
> > >> >Caused by: java.lang.Exception: Failed to parse value: null
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> > >> > at
> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
> > >> >le
> > >> >Pge
> > >> >Con
> > >> >figBuilder.java:77)
> > >> > ... 6 more
> > >> >Caused by: java.lang.NullPointerException
> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> > >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> > >> > at
> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR
> > >> >ep
> > >> >lac
> > >> >eme
> > >> >nt(PathUtils.java:321)
> > >> > at
> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P
> > >> >at
> > >> >hUt
> > >> >ils
> > >> >.java:96)
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> > >> > ... 8 more
> > >> >
> > >> >org.apache.oodt.cas.workflow.structs.exceptions.WorkflowTaskInstan
> > >> >ce
> > >> >Exc
> > >> >ept
> > >> >ion: PGETask FAILED!!! : Failed to build PgeConfig : Failed to
> > >> >parse
> > >> >value: null
> > >> > at
> > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:150)
> > >> > at
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >.ex
> > >> >ecu
> > >> >teTaskLocally(IterativeWorkflowProcessorThread.java:574)
> > >> > at
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >.ru
> > >> >n(I
> > >> >terativeWorkflowProcessorThread.java:321)
> > >> > at
> > >> >EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> > >>Source)
> > >> > at java.lang.Thread.run(Thread.java:745)
> > >> >Caused by: java.io.IOException: Failed to build PgeConfig : Failed
> > >> >to parse value: null
> > >> > at
> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
> > >> >le
> > >> >Pge
> > >> >Con
> > >> >figBuilder.java:87)
> > >> > at
> > >> >org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskIns
> > >> >ta
> > >> >nce
> > >> >.ja
> > >> >va:230)
> > >> > at
> > >> >org.apache.oodt.cas.pge.PGETaskInstance.run(PGETaskInstance.java:123)
> > >> > ... 4 more
> > >> >Caused by: java.lang.Exception: Failed to parse value: null
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:501)
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:480)
> > >> > at
> > >> >org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFi
> > >> >le
> > >> >Pge
> > >> >Con
> > >> >figBuilder.java:77)
> > >> > ... 6 more
> > >> >Caused by: java.lang.NullPointerException
> > >> > at java.util.regex.Matcher.getTextLength(Matcher.java:1234)
> > >> > at java.util.regex.Matcher.reset(Matcher.java:308)
> > >> > at java.util.regex.Matcher.<init>(Matcher.java:228)
> > >> > at java.util.regex.Pattern.matcher(Pattern.java:1088)
> > >> > at
> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicDateToMillisR
> > >> >ep
> > >> >lac
> > >> >eme
> > >> >nt(PathUtils.java:321)
> > >> > at
> > >> >org.apache.oodt.cas.metadata.util.PathUtils.doDynamicReplacement(P
> > >> >at
> > >> >hUt
> > >> >ils
> > >> >.java:96)
> > >> > at
> > >>org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:488)
> > >> > ... 8 more
> > >> >Oct 10, 2014 5:54:13 PM
> > >> >org.apache.oodt.cas.workflow.engine.IterativeWorkflowProcessorThre
> > >> >ad
> > >> >executeTaskLocally
> > >> >WARNING: Exception executing task: [feiCrawlerTaskName] locally:
> > >>Message:
> > >> >PGETask FAILED!!! : Failed to build PgeConfig : Failed to parse value:
> > >> >null
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >Content of tasks.xml
> > >> >
> > >> >
> > >> ><cas:tasks xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">
> > >> > <task id="urn:oodt:feiCrawlerTaskId" name="feiCrawlerTaskName"
> > >> >class="org.apache.oodt.cas.pge.StdPGETaskInstance">
> > >> > <conditions/>
> > >> >
> > >> > <configuration>
> > >> > <property name="PGETask/Name" value="feiCrawlerTaskName"/>
> > >> > <property name="PGETask/ConfigFilePath"
> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> > >> >de plo y/e xtensions/config/fei-crawler-pge-config.xml"/>
> > >> > <property name="PGETask/DumpMetadata" value="true"/>
> > >> > <property name="PGETask/WorkflowManagerUrl"
> > >> >value="http://localhost:9001" />
> > >> > <property name="PGETask/Query/FileManagerUrl"
> > >> >value="http://localhost:9000" />
> > >> > <property name="PGETask/Ingest/FileManagerUrl"
> > >> >value="http://localhost:9000"/>
> > >> >
> > >> > <property name="PGETask/Query/ClientTransferServiceFactory"
> > >> >value="org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferF
> > >> >ac
> > >> >tor
> > >> >y"/
> > >> >>
> > >> > <property name="PGETask/Ingest/CrawlerConfigFile"
> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> > >> >de
> > >> >plo
> > >> >y/c
> > >> >rawler/policy/crawler-config.xml"/>
> > >> > <property name="PGETask/Ingest/MimeExtractorRepo"
> > >> >value="file:/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-
> > >> >de plo y/e xtensions/policy/mime-extractor-map.xml"/>
> > >> > <property name="PGETask/Ingest/ActionIds"
> > >> >value="MoveFileToLevel0Dir"/>
> > >> > <property name="PGE_HOME"
> > >>
> > >>>value="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/pge"
> > >>>/>
> > >> > </configuration>
> > >> >
> > >> > <requiredMetFields/>
> > >> > </task>
> > >> ></cas:tasks>
> > >> >
> > >> >
> > >> >Contents of fei-crawler-pge-config.xml
> > >> >
> > >> >
> > >> ><pgeConfig>
> > >> >
> > >> > <!-- How to run the PGE -->
> > >> > <!-just echoing the current directory to a file so I can see if
> > >> >this thing ever starts -->
> > >> > <exe
> > >> >dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/
> > >> >da
> > >> >ta/
> > >> >pge
> > >> >/jobs" shell="/bin/sh">
> > >> > <cmd>echo "Current Working Directory is `pwd`" >
> > >> >/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy/logs/
> > >> >pg
> > >> >e.l
> > >> >og<
> > >> >/cmd>
> > >> > </exe>
> > >> >
> > >> > <!-- Files to ingest -->
> > >> > <output>
> > >> > <!-- trying this approach. Telling PGE there is output files
> > >> >should invoke the crawler. -->
> > >> >
> > >> ><dir="/homes/malldva1/project/jedi/users/jedi-pipeline/oodt-deploy
> > >> >/d
> > >> >ata
> > >> >/st
> > >> >aging"/>
> > >> > </output>
> > >> >
> > >> > <!-- Custom metadata to add to output files -->
> > >> > <customMetadata>
> > >> >
> > >> > <!-- each of these directories exist -->
> > >> > <metadata key="JobDir" val="[OODT_HOME]/data/pge/jobs"
> > >> >envReplace="true"/>
> > >> > <metadata key="JobInputDir" val="[JobDir]/input"/>
> > >> > <metadata key="JobOutputDir" val="[JobDir]/output"/>
> > >> > <metadata key="JobLogDir" val="[JobDir]/logs"/>
> > >> >
> > >> > </customMetadata>
> > >> ></pgeConfig>
> > >> >
> > >> >Valerie A. Mallder
> > >> >
> > >> >New Horizons Deputy Mission System Engineer The Johns Hopkins
> > >> >University/Applied Physics Laboratory
> > >> >11100 Johns Hopkins Rd (MS 23-282), Laurel, MD 20723
> > >> >240-228-7846 (Office) 410-504-2233 (Blackberry)
> > >> >
> > >
> >