Re: catalog queries in pge config files and filemgr-client problem

[Lewis John Mcgibbney <le...@gmail.com>]

Grand
Apologies for incorrect advice on logging Level. Log4j Vs.
java.util.Logging I think!

On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie <
Valerie.Mallder@jhuapl.edu> wrote:

> Hi Lewis,
>
> Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> recognized level. But, I used FINEST and added more log message and
> eventually found that I had run into the issue that has already been
> documented in OODT-854 which is supposed to be fixed in 0.10.  It has to do
> with where I was putting my policy files for the filemgr.  It sounds like
> the fix for OODT-854 will solve my problem.  Once I put all of my policy
> files in FILEMGR_HOME/policy/core, the catalog query started working.
>
> Thanks,
> Val
>
>
>
>
> > -----Original Message-----
> > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> > Sent: Thursday, August 27, 2015 8:07 PM
> > To: dev@oodt.apache.org
> > Subject: Re: catalog queries in pge config files and filemgr-client
> problem
> >
> > Hi Val,
> > Wise idea to start debugging this from the filemgr-client.
> > Can you please try changing logging levels in the following file to DEBUG
> >
> >
> https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/logging.pro
> > perties#L40-L74
> >
> > If you try your queries again then hopefully you should see some
> > patterns/indications within FileMgr logs to indicate what is going on.
> >
> >
> > On Wednesday, August 26, 2015, Mallder, Valerie <
> Valerie.Mallder@jhuapl.edu>
> > wrote:
> >
> > > Hi All,
> > >
> > > So, I'm phasing off of the New Horizons project for now and I am back
> > > to my science data pipeline work.  I am in the process of trying to
> > > pick up where I left off a few months ago - but with a twist J  I
> > > graduated from RADiX and now I am trying to get my pipeline system set
> > > up to run with the 'regular' OODT.  I have made extremely good
> > > progress so far. I have pretty much replicated all of the
> > > functionality of filemgr and workflow manager that I had been using
> > > with RADiX, with the exception of one thing that doesn't seem to be
> > > working the way it used too.  And I'm hoping someone out there can
> give me
> > some ideas on how to fix it.
> > >
> > > I have a catalog query in one of my PGE config files. I copied the
> > > syntax for the query statement directly from one of Chris' PGE config
> > > files
> > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > >
> > > Here is Chris' statement:
> > >
> > >     <metadata key="InputFiles"
> > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > RatLog}"/>
> > >
> > > Here is my statement:
> > >
> > >      <metadata key="InitialDirList"
> > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductReceived
> > > Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime FROM
> > > ScienceFile}"/>
> > >
> > > And here are the results from the query:
> > >
> > >
> > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filenam
> > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Filen
> > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > ocation/$Filename,$FileLocation/$Filename,
> > >
> > > And those results are not exactly what I wanted J
> > >
> > > So I ran some experiments with ./filemgr-client on the command line to
> > > make sure the files I expected to be in the catalog were in fact in
> > > the catalog.
> > >
> > > The following operation worked just as expected.  The expected number
> > > of products were listed.
> > > ./filemgr-client --url http://localhost:9016 --operation
> > > --getFirstPage --productTypeName ScienceFile
> > >
> > > However, this query gave some unexpected results. And this may be
> > > related the problem with the query in my PGE config file, but I am not
> sure.
> > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > >
> > > That query only outputs several blank lines to the screen and nothing
> else.
> > >
> > > I am not very familiar with all of the catalog related code yet so I
> > > am not sure where to start looking for the problem.  If you know this
> > > area of the code well and have some debugging ideas, please let me
> know.
> > >
> > > Thanks so much!
> > >
> > > Val
> > >
> > >
>



-- 
*Lewis*

RE: catalog queries in pge config files and filemgr-client problem

["Mallder, Valerie" <Va...@jhuapl.edu>]

No problem!


Valerie A. Mallder
New Horizons Deputy Mission System Engineer
Johns Hopkins University/Applied Physics Laboratory


> -----Original Message-----
> From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> Sent: Monday, August 31, 2015 6:02 PM
> To: dev@oodt.apache.org
> Subject: Re: catalog queries in pge config files and filemgr-client problem
> 
> Grand
> Apologies for incorrect advice on logging Level. Log4j Vs.
> java.util.Logging I think!
> 
> On Mon, Aug 31, 2015 at 2:41 PM, Mallder, Valerie < Valerie.Mallder@jhuapl.edu>
> wrote:
> 
> > Hi Lewis,
> >
> > Thanks Lewis, I tried setting it to DEBUG, that it told me it wasn't a
> > recognized level. But, I used FINEST and added more log message and
> > eventually found that I had run into the issue that has already been
> > documented in OODT-854 which is supposed to be fixed in 0.10.  It has
> > to do with where I was putting my policy files for the filemgr.  It
> > sounds like the fix for OODT-854 will solve my problem.  Once I put
> > all of my policy files in FILEMGR_HOME/policy/core, the catalog query started
> working.
> >
> > Thanks,
> > Val
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Lewis John Mcgibbney [mailto:lewis.mcgibbney@gmail.com]
> > > Sent: Thursday, August 27, 2015 8:07 PM
> > > To: dev@oodt.apache.org
> > > Subject: Re: catalog queries in pge config files and filemgr-client
> > problem
> > >
> > > Hi Val,
> > > Wise idea to start debugging this from the filemgr-client.
> > > Can you please try changing logging levels in the following file to
> > > DEBUG
> > >
> > >
> > https://github.com/apache/oodt/blob/trunk/filemgr/src/main/resources/l
> > ogging.pro
> > > perties#L40-L74
> > >
> > > If you try your queries again then hopefully you should see some
> > > patterns/indications within FileMgr logs to indicate what is going on.
> > >
> > >
> > > On Wednesday, August 26, 2015, Mallder, Valerie <
> > Valerie.Mallder@jhuapl.edu>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > So, I'm phasing off of the New Horizons project for now and I am
> > > > back to my science data pipeline work.  I am in the process of
> > > > trying to pick up where I left off a few months ago - but with a
> > > > twist J  I graduated from RADiX and now I am trying to get my
> > > > pipeline system set up to run with the 'regular' OODT.  I have
> > > > made extremely good progress so far. I have pretty much replicated
> > > > all of the functionality of filemgr and workflow manager that I
> > > > had been using with RADiX, with the exception of one thing that
> > > > doesn't seem to be working the way it used too.  And I'm hoping
> > > > someone out there can
> > give me
> > > some ideas on how to fix it.
> > > >
> > > > I have a catalog query in one of my PGE config files. I copied the
> > > > syntax for the query statement directly from one of Chris' PGE
> > > > config files
> > > > (PgeConfig_RatAggregator.xml) in the 'drat-master' application.
> > > >
> > > > Here is Chris' statement:
> > > >
> > > >     <metadata key="InputFiles"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM RatLog}"/>
> > > >
> > > > Here is my statement:
> > > >
> > > >      <metadata key="InitialDirList"
> > > > val="SQL(FORMAT='$FileLocation/$Filename',SORT_BY='CAS.ProductRece
> > > > ived Time'){SELECT FileLocation,Filename,CAS.ProductReceivedTime
> > > > FROM ScienceFile}"/>
> > > >
> > > > And here are the results from the query:
> > > >
> > > >
> > > > $FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$Fil
> > > > enam
> > > > e,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/$F
> > > > ilen
> > > > ame,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocation/
> > > > $Fil
> > > > ename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocatio
> > > > n/$F
> > > > ilename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLocat
> > > > ion/
> > > > $Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileLoc
> > > > atio
> > > > n/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$FileL
> > > > ocat
> > > > ion/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$Fil
> > > > eLoc
> > > > ation/$Filename,$FileLocation/$Filename,$FileLocation/$Filename,$F
> > > > ileL ocation/$Filename,$FileLocation/$Filename,
> > > >
> > > > And those results are not exactly what I wanted J
> > > >
> > > > So I ran some experiments with ./filemgr-client on the command
> > > > line to make sure the files I expected to be in the catalog were
> > > > in fact in the catalog.
> > > >
> > > > The following operation worked just as expected.  The expected
> > > > number of products were listed.
> > > > ./filemgr-client --url http://localhost:9016 --operation
> > > > --getFirstPage --productTypeName ScienceFile
> > > >
> > > > However, this query gave some unexpected results. And this may be
> > > > related the problem with the query in my PGE config file, but I am
> > > > not
> > sure.
> > > > ./filemgr-client --url http://localhost:9016 --operation -sqlQuery
> > > > -query "SELECT FileLocation,Filename FROM ScienceFile"
> > > >
> > > > That query only outputs several blank lines to the screen and
> > > > nothing
> > else.
> > > >
> > > > I am not very familiar with all of the catalog related code yet so
> > > > I am not sure where to start looking for the problem.  If you know
> > > > this area of the code well and have some debugging ideas, please
> > > > let me
> > know.
> > > >
> > > > Thanks so much!
> > > >
> > > > Val
> > > >
> > > >
> >
> 
> 
> 
> --
> *Lewis*