You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by "Mallder, Valerie" <Va...@jhuapl.edu> on 2015/08/27 02:15:40 UTC
catalog queries in pge config files and filemgr-client problem
Hi All,
So, I'm phasing off of the New Horizons project for now and I am back to my science data pipeline work. I am in the process of trying to pick up where I left off a few months ago - but with a twist J I graduated from RADiX and now I am trying to get my pipeline system set up to run with the 'regular' OODT. I have made extremely good progress so far. I have pretty much replicated all of the functionality of filemgr and workflow manager that I had been using with RADiX, with the exception of one thing that doesn't seem to be working the way it used too. And I'm hoping someone out there can give me some ideas on how to fix it.
I have a catalog query in one of my PGE config files. I copied the syntax for the query statement directly from one of Chris' PGE config files (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
Here is Chris' statement:
<metadata key="InputFiles" val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/>
Here is my statement:
<metadata key="InitialDirList" val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/>
And here are the results from the query:
$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,
And those results are not exactly what I wanted J
So I ran some experiments with ./filemgr-client on the command line to make sure the files I expected to be in the catalog were in fact in the catalog.
The following operation worked just as expected. The expected number of products were listed.
./filemgr-client --url http://localhost:9016 --operation --getFirstPage --productTypeName ScienceFile
However, this query gave some unexpected results. And this may be related the problem with the query in my PGE config file, but I am not sure.
./filemgr-client --url http://localhost:9016 --operation -sqlQuery -query "SELECT FileLocation,Filename FROM ScienceFile"
That query only outputs several blank lines to the screen and nothing else.
I am not very familiar with all of the catalog related code yet so I am not sure where to start looking for the problem. If you know this area of the code well and have some debugging ideas, please let me know.
Thanks so much!
Val
RE: catalog queries in pge config files and filemgr-client problem
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
No problem!
Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory
> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Monday, August 31, 2015 6:02 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
>
> Grand
> Apologies for incorrect advice on logging Level. Log4j Vs.
> java.util.Logging I think!
>
> On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie < Valerie.Mallder@jhuapl.edu>
> wrote:
>
> > Hi Lewis,
> >
> > Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> > recognized level. But, I used FINEST and added more log message and
> > eventually found that I had run into the issue that has already been
> > documented in OODT-854 which is supposed to be fixed in 0.10. It has
> > to do with where I was putting my policy files for the filemgr. It
> > sounds like the fix for OODT-854 will solve my problem. Once I put
> > all of my policy files in FILEMGR_HOME/policy/core, the catalog query started
> working.
> >
> > Thanks,
> > Val
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> > > Sent: Thursday, August 27, 2015 8:07 PM
> > > To: dev@oodt.apache.org
> > > Subject: Re: catalog queries in pge config files and filemgr-client
> > problem
> > >
> > > Hi Val,
> > > Wise idea to start debugging this from the filemgr-client.
> > > Can you please try changing logging levels in the following file to
> > > DEBUG
> > >
> > >
> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/l
> > ogging.pro
> > > perties#L40-L74
> > >
> > > If you try your queries again then hopefully you should see some
> > > patterns/indications within FileMgr logs to indicate what is going on.
> > >
> > >
> > > On Wednesday, August 26, 2015, Mallder, Valerie <
> > Valerie.Mallder@jhuapl.edu>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > So, I'm phasing off of the New Horizons project for now and I am
> > > > back to my science data pipeline work. I am in the process of
> > > > trying to pick up where I left off a few months ago - but with a
> > > > twist J I graduated from RADiX and now I am trying to get my
> > > > pipeline system set up to run with the 'regular' OODT. I have
> > > > made extremely good progress so far. I have pretty much replicated
> > > > all of the functionality of filemgr and workflow manager that I
> > > > had been using with RADiX, with the exception of one thing that
> > > > doesn't seem to be working the way it used too. And I'm hoping
> > > > someone out there can
> > give me
> > > some ideas on how to fix it.
> > > >
> > > > I have a catalog query in one of my PGE config files. I copied the
> > > > syntax for the query statement directly from one of Chris' PGE
> > > > config files
> > > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > > >
> > > > Here is Chris' statement:
> > > >
> > > > <metadata key="InputFiles"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM RatLog}"/>
> > > >
> > > > Here is my statement:
> > > >
> > > > <metadata key="InitialDirList"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM ScienceFile}"/>
> > > >
> > > > And here are the results from the query:
> > > >
> > > >
> > > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > > enam
> > > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > > ilen
> > > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > > $Fil
> > > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > > n/$F
> > > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > > ion/
> > > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > > atio
> > > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > > ocat
> > > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
> > > > eLoc
> > > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
> > > > ileL ocation/$Filename,$FileLocation/$Filename,
> > > >
> > > > And those results are not exactly what I wanted J
> > > >
> > > > So I ran some experiments with ./filemgr-client on the command
> > > > line to make sure the files I expected to be in the catalog were
> > > > in fact in the catalog.
> > > >
> > > > The following operation worked just as expected. The expected
> > > > number of products were listed.
> > > > ./filemgr-client --url http://localhost:9016 --operation
> > > > --getFirstPage --productTypeName ScienceFile
> > > >
> > > > However, this query gave some unexpected results. And this may be
> > > > related the problem with the query in my PGE config file, but I am
> > > > not
> > sure.
> > > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > > >
> > > > That query only outputs several blank lines to the screen and
> > > > nothing
> > else.
> > > >
> > > > I am not very familiar with all of the catalog related code yet so
> > > > I am not sure where to start looking for the problem. If you know
> > > > this area of the code well and have some debugging ideas, please
> > > > let me
> > know.
> > > >
> > > > Thanks so much!
> > > >
> > > > Val
> > > >
> > > >
> >
>
>
>
> --
> *Lewis*
Re: catalog queries in pge config files and filemgr-client problem
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Grand
Apologies for incorrect advice on logging Level. Log4j Vs.
java.util.Logging I think!
On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie <
Valerie.Mallder@jhuapl.edu> wrote:
> Hi Lewis,
>
> Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> recognized level. But, I used FINEST and added more log message and
> eventually found that I had run into the issue that has already been
> documented in OODT-854 which is supposed to be fixed in 0.10. It has to do
> with where I was putting my policy files for the filemgr. It sounds like
> the fix for OODT-854 will solve my problem. Once I put all of my policy
> files in FILEMGR_HOME/policy/core, the catalog query started working.
>
> Thanks,
> Val
>
>
>
>
> > -----Original Message-----
> > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> > Sent: Thursday, August 27, 2015 8:07 PM
> > To: dev@oodt.apache.org
> > Subject: Re: catalog queries in pge config files and filemgr-client
> problem
> >
> > Hi Val,
> > Wise idea to start debugging this from the filemgr-client.
> > Can you please try changing logging levels in the following file to DEBUG
> >
> >
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.pro
> > perties#L40-L74
> >
> > If you try your queries again then hopefully you should see some
> > patterns/indications within FileMgr logs to indicate what is going on.
> >
> >
> > On Wednesday, August 26, 2015, Mallder, Valerie <
> Valerie.Mallder@jhuapl.edu>
> > wrote:
> >
> > > Hi All,
> > >
> > > So, I'm phasing off of the New Horizons project for now and I am back
> > > to my science data pipeline work. I am in the process of trying to
> > > pick up where I left off a few months ago - but with a twist J I
> > > graduated from RADiX and now I am trying to get my pipeline system set
> > > up to run with the 'regular' OODT. I have made extremely good
> > > progress so far. I have pretty much replicated all of the
> > > functionality of filemgr and workflow manager that I had been using
> > > with RADiX, with the exception of one thing that doesn't seem to be
> > > working the way it used too. And I'm hoping someone out there can
> give me
> > some ideas on how to fix it.
> > >
> > > I have a catalog query in one of my PGE config files. I copied the
> > > syntax for the query statement directly from one of Chris' PGE config
> > > files
> > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > >
> > > Here is Chris' statement:
> > >
> > > <metadata key="InputFiles"
> > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > RatLog}"/>
> > >
> > > Here is my statement:
> > >
> > > <metadata key="InitialDirList"
> > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > ScienceFile}"/>
> > >
> > > And here are the results from the query:
> > >
> > >
> > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filenam
> > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filen
> > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > ocation/$Filename,$FileLocation/$Filename,
> > >
> > > And those results are not exactly what I wanted J
> > >
> > > So I ran some experiments with ./filemgr-client on the command line to
> > > make sure the files I expected to be in the catalog were in fact in
> > > the catalog.
> > >
> > > The following operation worked just as expected. The expected number
> > > of products were listed.
> > > ./filemgr-client --url http://localhost:9016 --operation
> > > --getFirstPage --productTypeName ScienceFile
> > >
> > > However, this query gave some unexpected results. And this may be
> > > related the problem with the query in my PGE config file, but I am not
> sure.
> > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > >
> > > That query only outputs several blank lines to the screen and nothing
> else.
> > >
> > > I am not very familiar with all of the catalog related code yet so I
> > > am not sure where to start looking for the problem. If you know this
> > > area of the code well and have some debugging ideas, please let me
> know.
> > >
> > > Thanks so much!
> > >
> > > Val
> > >
> > >
>
--
*Lewis*
RE: catalog queries in pge config files and filemgr-client problem
Posted by "Mallder, Valerie" <Va...@jhuapl.edu>.
Hi Lewis,
Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a recognized level. But, I used FINEST and added more log message and eventually found that I had run into the issue that has already been documented in OODT-854 which is supposed to be fixed in 0.10. It has to do with where I was putting my policy files for the filemgr. It sounds like the fix for OODT-854 will solve my problem. Once I put all of my policy files in FILEMGR_HOME/policy/core, the catalog query started working.
Thanks,
Val
> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Thursday, August 27, 2015 8:07 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
>
> Hi Val,
> Wise idea to start debugging this from the filemgr-client.
> Can you please try changing logging levels in the following file to DEBUG
>
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.pro
> perties#L40-L74
>
> If you try your queries again then hopefully you should see some
> patterns/indications within FileMgr logs to indicate what is going on.
>
>
> On Wednesday, August 26, 2015, Mallder, Valerie <Va...@jhuapl.edu>
> wrote:
>
> > Hi All,
> >
> > So, I'm phasing off of the New Horizons project for now and I am back
> > to my science data pipeline work. I am in the process of trying to
> > pick up where I left off a few months ago - but with a twist J I
> > graduated from RADiX and now I am trying to get my pipeline system set
> > up to run with the 'regular' OODT. I have made extremely good
> > progress so far. I have pretty much replicated all of the
> > functionality of filemgr and workflow manager that I had been using
> > with RADiX, with the exception of one thing that doesn't seem to be
> > working the way it used too. And I'm hoping someone out there can give me
> some ideas on how to fix it.
> >
> > I have a catalog query in one of my PGE config files. I copied the
> > syntax for the query statement directly from one of Chris' PGE config
> > files
> > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> >
> > Here is Chris' statement:
> >
> > <metadata key="InputFiles"
> > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > RatLog}"/>
> >
> > Here is my statement:
> >
> > <metadata key="InitialDirList"
> > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > ScienceFile}"/>
> >
> > And here are the results from the query:
> >
> >
> > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filenam
> > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filen
> > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > ocation/$Filename,$FileLocation/$Filename,
> >
> > And those results are not exactly what I wanted J
> >
> > So I ran some experiments with ./filemgr-client on the command line to
> > make sure the files I expected to be in the catalog were in fact in
> > the catalog.
> >
> > The following operation worked just as expected. The expected number
> > of products were listed.
> > ./filemgr-client --url http://localhost:9016 --operation
> > --getFirstPage --productTypeName ScienceFile
> >
> > However, this query gave some unexpected results. And this may be
> > related the problem with the query in my PGE config file, but I am not sure.
> > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > -query "SELECT FileLocation,Filename FROM ScienceFile"
> >
> > That query only outputs several blank lines to the screen and nothing else.
> >
> > I am not very familiar with all of the catalog related code yet so I
> > am not sure where to start looking for the problem. If you know this
> > area of the code well and have some debugging ideas, please let me know.
> >
> > Thanks so much!
> >
> > Val
> >
> >
Re: catalog queries in pge config files and filemgr-client problem
Posted by "Mattmann, Chris A (3980)" <ch...@jpl.nasa.gov>.
Val, I would inspect your file manager catalog. Can you use the OPSUI
or some other browser (Lucene) to see the metadata that is recorded
for your files? I think this is part of the problem - from the query
the only reason that $FileLocation/$Filename wouldn’t be resolved in
your queries is that they aren’t recorded as metadata fields for your
products.
Can you check?
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
-----Original Message-----
From: Lewis John Mcgibbney <le...@gmail.com>
Reply-To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Date: Thursday, August 27, 2015 at 5:07 PM
To: "dev@oodt.apache.org" <de...@oodt.apache.org>
Subject: Re: catalog queries in pge config files and filemgr-client problem
>Hi Val,
>Wise idea to start debugging this from the filemgr-client.
>Can you please try changing logging levels in the following file to DEBUG
>
>https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/loggi
>ng.properties#L40-L74
>
>If you try your queries again then hopefully you should see some
>patterns/indications within FileMgr logs to indicate what is going on.
>
>
>On Wednesday, August 26, 2015, Mallder, Valerie
><Va...@jhuapl.edu>
>wrote:
>
>> Hi All,
>>
>> So, I'm phasing off of the New Horizons project for now and I am back to
>> my science data pipeline work. I am in the process of trying to pick up
>> where I left off a few months ago - but with a twist J I graduated from
>> RADiX and now I am trying to get my pipeline system set up to run with
>>the
>> 'regular' OODT. I have made extremely good progress so far. I have
>>pretty
>> much replicated all of the functionality of filemgr and workflow manager
>> that I had been using with RADiX, with the exception of one thing that
>> doesn't seem to be working the way it used too. And I'm hoping someone
>>out
>> there can give me some ideas on how to fix it.
>>
>> I have a catalog query in one of my PGE config files. I copied the
>>syntax
>> for the query statement directly from one of Chris' PGE config files
>> (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
>>
>> Here is Chris' statement:
>>
>> <metadata key="InputFiles"
>>
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/>
>>
>> Here is my statement:
>>
>> <metadata key="InitialDirList"
>>
>>val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTim
>>e'){SELECT
>> FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/>
>>
>> And here are the results from the query:
>>
>>
>>
>>$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$
>>FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
>>ileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fi
>>leLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
>>eLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$File
>>Location/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
>>ocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLo
>>cation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
>>ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoca
>>tion/$Filename,
>>
>> And those results are not exactly what I wanted J
>>
>> So I ran some experiments with ./filemgr-client on the command line to
>> make sure the files I expected to be in the catalog were in fact in the
>> catalog.
>>
>> The following operation worked just as expected. The expected number of
>> products were listed.
>> ./filemgr-client --url http://localhost:9016 --operation --getFirstPage
>> --productTypeName ScienceFile
>>
>> However, this query gave some unexpected results. And this may be
>>related
>> the problem with the query in my PGE config file, but I am not sure.
>> ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
>>-query
>> "SELECT FileLocation,Filename FROM ScienceFile"
>>
>> That query only outputs several blank lines to the screen and nothing
>>else.
>>
>> I am not very familiar with all of the catalog related code yet so I am
>> not sure where to start looking for the problem. If you know this area
>>of
>> the code well and have some debugging ideas, please let me know.
>>
>> Thanks so much!
>>
>> Val
>>
>>
Re: catalog queries in pge config files and filemgr-client problem
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Hi Val,
Wise idea to start debugging this from the filemgr-client.
Can you please try changing logging levels in the following file to DEBUG
https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.properties#L40-L74
If you try your queries again then hopefully you should see some
patterns/indications within FileMgr logs to indicate what is going on.
On Wednesday, August 26, 2015, Mallder, Valerie <Va...@jhuapl.edu>
wrote:
> Hi All,
>
> So, I'm phasing off of the New Horizons project for now and I am back to
> my science data pipeline work. I am in the process of trying to pick up
> where I left off a few months ago - but with a twist J I graduated from
> RADiX and now I am trying to get my pipeline system set up to run with the
> 'regular' OODT. I have made extremely good progress so far. I have pretty
> much replicated all of the functionality of filemgr and workflow manager
> that I had been using with RADiX, with the exception of one thing that
> doesn't seem to be working the way it used too. And I'm hoping someone out
> there can give me some ideas on how to fix it.
>
> I have a catalog query in one of my PGE config files. I copied the syntax
> for the query statement directly from one of Chris' PGE config files
> (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
>
> Here is Chris' statement:
>
> <metadata key="InputFiles"
> val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT
> FileLocation,Filename,CAS.ProductReceivedTime FROM RatLog}"/>
>
> Here is my statement:
>
> <metadata key="InitialDirList"
> val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceivedTime'){SELECT
> FileLocation,Filename,CAS.ProductReceivedTime FROM ScienceFile}"/>
>
> And here are the results from the query:
>
>
> $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,
>
> And those results are not exactly what I wanted J
>
> So I ran some experiments with ./filemgr-client on the command line to
> make sure the files I expected to be in the catalog were in fact in the
> catalog.
>
> The following operation worked just as expected. The expected number of
> products were listed.
> ./filemgr-client --url http://localhost:9016 --operation --getFirstPage
> --productTypeName ScienceFile
>
> However, this query gave some unexpected results. And this may be related
> the problem with the query in my PGE config file, but I am not sure.
> ./filemgr-client --url http://localhost:9016 --operation -sqlQuery -query
> "SELECT FileLocation,Filename FROM ScienceFile"
>
> That query only outputs several blank lines to the screen and nothing else.
>
> I am not very familiar with all of the catalog related code yet so I am
> not sure where to start looking for the problem. If you know this area of
> the code well and have some debugging ideas, please let me know.
>
> Thanks so much!
>
> Val
>
>