You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Sheryl John <sh...@gmail.com> on 2011/09/15 04:03:33 UTC
PGETask Metadata
Hi,
I have defined some key-val pairs for a file( say Output.csv) in a
metout-config.xml for my PGETask Workflow. However, after executing the
workflow, the met-config.xml is not creating a Output.csv.cas file.
I want to be able to use the above keys/metadata later on in an SQL-like
query from the pgeconfig file. For example, if I've defined 'RecordID' as a
key in the metout-config.xml, I would want to use this metadata in the
following query:
SQL(FORMAT='$FileLocation/$Filename'){ SELECT
FileLocation,Filename,ISMTable,*RecordID* FROM ISMRawData WHERE ISMTable =
'Chartevents' AND *RecordID* = "PID"} The others keys included in the
query above are elements and product-types that were defined during
ingestion in the File Manager.
At present, the task fails to parse the above query when I run the workflow.
Is this because metout-config is not creating the Output.cas file? And,
what is the best way to specify metadata files for a group of files or for a
folder?
Thanks,
Sheryl
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Sheryl.
OK, that tells me that the met you are expecting to be cataloged isn't.
Can you please post the relevant portions of your product-types.xml,
product-type-element-map.xml, and elements.xml to the list? You might
want to use pastebin, b/c I don't think the list takes attachments.
Cheers,
Chris
On Sep 15, 2011, at 10:19 PM, Sheryl John wrote:
> I tried out for a couple of files, and the metdump alias is not dumping out any metadata for the respective ProductIds.
>
> I also tried dumping out metadata for other files for which I've not specified metout-configs, and they seem to be empty too.
> The fmquery works fine and it dumps out all the ProductIds.
>
> Thanks,
>
> On Thu, Sep 15, 2011 at 9:08 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> Are you sure that RecordID is being recorded for the particular products that you are trying to look up?
>
> One way to find this out:
>
> 1. get the product IDs back for the products you are interested in. One way to do this would be
> to remove RecordID from your query as you mention below and then add CAS.ProductId to the
> SELECT list, and to change FORMAT='$FileLocation...' to FORMAT='$CAS.ProductId'.
>
> 2. Then, for each of the product IDs, check out OODT-306 [1] and use the metdump alias to dump
> out the metadata for each product ID to the current working directory. Then, inspect that .met file.
> Is RecordID recorded?
>
> Let's start there.
>
> Cheers,
> Chris
>
> [1] http://issues.apache.org/jira/browse/OODT-306
>
> On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I am not clear about the metadata in .met files generated by the met-config.xml files. Particularly, where do they fall in the product-type and element mappings?
> >
> > Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are mapped to the product-type, ISMRawData and elements that are mapped to the GenericFile product-type.
> >
> > Except RecordID, which was defined as a key in the metout-config file and I'm assuming that the parsing error is because the RecordID is not mapped to any product-type. This query works and pulls all the files I need when I remove RecordID from the query. Again, since I'm not clear about metadata mapping , it's highly likely that I'm wrong about this.
> >
> > Thanks,
> > On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> > hey Sheryl,
> >
> > I think there is a misunderstanding on how the met-config.xml files are used... the metadata in these files are not added to your workflow context metadata... they are only used to create the *.met files for the data file for filemgr ingest... about your query I'm not quite clear on how you are wanting go use it... it kinda seems like you want to dynamically replace out RecordIO... in which case you're gonna need it to look more like RecordIO = '[PID]'
> >
> > -brian
> >
> >
> > On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
> >>
> >> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
> >> I guess I have revise my workflow logic and metadata use before getting back to this query.
> >>
> >> Thanks!
> >>
> >>
> >>
> >> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> >> Hi Sheryl,
> >>
> >> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
> >>
> >> Try using single quotes on both and see if that fixes it.
> >>
> >> Thanks!
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> >>
> >> > Hi,
> >> >
> >> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
> >> >
> >> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
> >> >
> >> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
> >> >
> >> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
> >> >
> >> > Thanks,
> >> > Sheryl
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasagov
> >>
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >> --
> >> -Sheryl
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Sheryl.
OK, that tells me that the met you are expecting to be cataloged isn't.
Can you please post the relevant portions of your product-types.xml,
product-type-element-map.xml, and elements.xml to the list? You might
want to use pastebin, b/c I don't think the list takes attachments.
Cheers,
Chris
On Sep 15, 2011, at 10:19 PM, Sheryl John wrote:
> I tried out for a couple of files, and the metdump alias is not dumping out any metadata for the respective ProductIds.
>
> I also tried dumping out metadata for other files for which I've not specified metout-configs, and they seem to be empty too.
> The fmquery works fine and it dumps out all the ProductIds.
>
> Thanks,
>
> On Thu, Sep 15, 2011 at 9:08 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> Are you sure that RecordID is being recorded for the particular products that you are trying to look up?
>
> One way to find this out:
>
> 1. get the product IDs back for the products you are interested in. One way to do this would be
> to remove RecordID from your query as you mention below and then add CAS.ProductId to the
> SELECT list, and to change FORMAT='$FileLocation...' to FORMAT='$CAS.ProductId'.
>
> 2. Then, for each of the product IDs, check out OODT-306 [1] and use the metdump alias to dump
> out the metadata for each product ID to the current working directory. Then, inspect that .met file.
> Is RecordID recorded?
>
> Let's start there.
>
> Cheers,
> Chris
>
> [1] http://issues.apache.org/jira/browse/OODT-306
>
> On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I am not clear about the metadata in .met files generated by the met-config.xml files. Particularly, where do they fall in the product-type and element mappings?
> >
> > Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are mapped to the product-type, ISMRawData and elements that are mapped to the GenericFile product-type.
> >
> > Except RecordID, which was defined as a key in the metout-config file and I'm assuming that the parsing error is because the RecordID is not mapped to any product-type. This query works and pulls all the files I need when I remove RecordID from the query. Again, since I'm not clear about metadata mapping , it's highly likely that I'm wrong about this.
> >
> > Thanks,
> > On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> > hey Sheryl,
> >
> > I think there is a misunderstanding on how the met-config.xml files are used... the metadata in these files are not added to your workflow context metadata... they are only used to create the *.met files for the data file for filemgr ingest... about your query I'm not quite clear on how you are wanting go use it... it kinda seems like you want to dynamically replace out RecordIO... in which case you're gonna need it to look more like RecordIO = '[PID]'
> >
> > -brian
> >
> >
> > On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
> >>
> >> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
> >> I guess I have revise my workflow logic and metadata use before getting back to this query.
> >>
> >> Thanks!
> >>
> >>
> >>
> >> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> >> Hi Sheryl,
> >>
> >> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
> >>
> >> Try using single quotes on both and see if that fixes it.
> >>
> >> Thanks!
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> >>
> >> > Hi,
> >> >
> >> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
> >> >
> >> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
> >> >
> >> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
> >> >
> >> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
> >> >
> >> > Thanks,
> >> > Sheryl
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasagov
> >>
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >> --
> >> -Sheryl
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
I tried out for a couple of files, and the metdump alias is not dumping out
any metadata for the respective ProductIds.
I also tried dumping out metadata for other files for which I've not
specified metout-configs, and they seem to be empty too.
The fmquery works fine and it dumps out all the ProductIds.
Thanks,
On Thu, Sep 15, 2011 at 9:08 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> Are you sure that RecordID is being recorded for the particular products
> that you are trying to look up?
>
> One way to find this out:
>
> 1. get the product IDs back for the products you are interested in. One way
> to do this would be
> to remove RecordID from your query as you mention below and then add
> CAS.ProductId to the
> SELECT list, and to change FORMAT='$FileLocation...' to
> FORMAT='$CAS.ProductId'.
>
> 2. Then, for each of the product IDs, check out OODT-306 [1] and use the
> metdump alias to dump
> out the metadata for each product ID to the current working directory.
> Then, inspect that .met file.
> Is RecordID recorded?
>
> Let's start there.
>
> Cheers,
> Chris
>
> [1] http://issues.apache.org/jira/browse/OODT-306
>
> On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I am not clear about the metadata in .met files generated by the
> met-config.xml files. Particularly, where do they fall in the product-type
> and element mappings?
> >
> > Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are
> mapped to the product-type, ISMRawData and elements that are mapped to the
> GenericFile product-type.
> >
> > Except RecordID, which was defined as a key in the metout-config file
> and I'm assuming that the parsing error is because the RecordID is not
> mapped to any product-type. This query works and pulls all the files I need
> when I remove RecordID from the query. Again, since I'm not clear about
> metadata mapping , it's highly likely that I'm wrong about this.
> >
> > Thanks,
> > On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> > hey Sheryl,
> >
> > I think there is a misunderstanding on how the met-config.xml files are
> used... the metadata in these files are not added to your workflow context
> metadata... they are only used to create the *.met files for the data file
> for filemgr ingest... about your query I'm not quite clear on how you are
> wanting go use it... it kinda seems like you want to dynamically replace out
> RecordIO... in which case you're gonna need it to look more like RecordIO =
> '[PID]'
> >
> > -brian
> >
> >
> > On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for pointing that out. I was working on a different query and
> similar to the one I described. Though I've used only single quotes now, I'm
> still getting the same parsing error.
> >>
> >> But, I think what I intended to do with context metadata ( for my
> workflow) doesn't make sense for a single file product and rather, would
> apply for a group of files.
> >> I guess I have revise my workflow logic and metadata use before getting
> back to this query.
> >>
> >> Thanks!
> >>
> >>
> >>
> >> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
> >> Hi Sheryl,
> >>
> >> I notice you're using double quotes on the RecordID = "PID", yet you are
> using single quotes on ISMTable = 'Chartevents'.
> >>
> >> Try using single quotes on both and see if that fixes it.
> >>
> >> Thanks!
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> >>
> >> > Hi,
> >> >
> >> > I have defined some key-val pairs for a file( say Output.csv) in a
> metout-config.xml for my PGETask Workflow. However, after executing the
> workflow, the met-config.xml is not creating a Output.csv.cas file.
> >> >
> >> > I want to be able to use the above keys/metadata later on in an
> SQL-like query from the pgeconfig file. For example, if I've defined
> 'RecordID' as a key in the metout-config.xml, I would want to use this
> metadata in the following query:
> >> >
> >> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
> above are elements and product-types that were defined during ingestion in
> the File Manager.
> >> >
> >> > At present, the task fails to parse the above query when I run the
> workflow. Is this because metout-config is not creating the Output.cas file?
> And, what is the best way to specify metadata files for a group of files or
> for a folder?
> >> >
> >> > Thanks,
> >> > Sheryl
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasagov
> >>
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >> --
> >> -Sheryl
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
I tried out for a couple of files, and the metdump alias is not dumping out
any metadata for the respective ProductIds.
I also tried dumping out metadata for other files for which I've not
specified metout-configs, and they seem to be empty too.
The fmquery works fine and it dumps out all the ProductIds.
Thanks,
On Thu, Sep 15, 2011 at 9:08 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Sheryl,
>
> Are you sure that RecordID is being recorded for the particular products
> that you are trying to look up?
>
> One way to find this out:
>
> 1. get the product IDs back for the products you are interested in. One way
> to do this would be
> to remove RecordID from your query as you mention below and then add
> CAS.ProductId to the
> SELECT list, and to change FORMAT='$FileLocation...' to
> FORMAT='$CAS.ProductId'.
>
> 2. Then, for each of the product IDs, check out OODT-306 [1] and use the
> metdump alias to dump
> out the metadata for each product ID to the current working directory.
> Then, inspect that .met file.
> Is RecordID recorded?
>
> Let's start there.
>
> Cheers,
> Chris
>
> [1] http://issues.apache.org/jira/browse/OODT-306
>
> On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I am not clear about the metadata in .met files generated by the
> met-config.xml files. Particularly, where do they fall in the product-type
> and element mappings?
> >
> > Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are
> mapped to the product-type, ISMRawData and elements that are mapped to the
> GenericFile product-type.
> >
> > Except RecordID, which was defined as a key in the metout-config file
> and I'm assuming that the parsing error is because the RecordID is not
> mapped to any product-type. This query works and pulls all the files I need
> when I remove RecordID from the query. Again, since I'm not clear about
> metadata mapping , it's highly likely that I'm wrong about this.
> >
> > Thanks,
> > On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> > hey Sheryl,
> >
> > I think there is a misunderstanding on how the met-config.xml files are
> used... the metadata in these files are not added to your workflow context
> metadata... they are only used to create the *.met files for the data file
> for filemgr ingest... about your query I'm not quite clear on how you are
> wanting go use it... it kinda seems like you want to dynamically replace out
> RecordIO... in which case you're gonna need it to look more like RecordIO =
> '[PID]'
> >
> > -brian
> >
> >
> > On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
> >
> >> Hi Chris,
> >>
> >> Thanks for pointing that out. I was working on a different query and
> similar to the one I described. Though I've used only single quotes now, I'm
> still getting the same parsing error.
> >>
> >> But, I think what I intended to do with context metadata ( for my
> workflow) doesn't make sense for a single file product and rather, would
> apply for a group of files.
> >> I guess I have revise my workflow logic and metadata use before getting
> back to this query.
> >>
> >> Thanks!
> >>
> >>
> >>
> >> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
> >> Hi Sheryl,
> >>
> >> I notice you're using double quotes on the RecordID = "PID", yet you are
> using single quotes on ISMTable = 'Chartevents'.
> >>
> >> Try using single quotes on both and see if that fixes it.
> >>
> >> Thanks!
> >>
> >> Cheers,
> >> Chris
> >>
> >>
> >> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> >>
> >> > Hi,
> >> >
> >> > I have defined some key-val pairs for a file( say Output.csv) in a
> metout-config.xml for my PGETask Workflow. However, after executing the
> workflow, the met-config.xml is not creating a Output.csv.cas file.
> >> >
> >> > I want to be able to use the above keys/metadata later on in an
> SQL-like query from the pgeconfig file. For example, if I've defined
> 'RecordID' as a key in the metout-config.xml, I would want to use this
> metadata in the following query:
> >> >
> >> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
> above are elements and product-types that were defined during ingestion in
> the File Manager.
> >> >
> >> > At present, the task fails to parse the above query when I run the
> workflow. Is this because metout-config is not creating the Output.cas file?
> And, what is the best way to specify metadata files for a group of files or
> for a folder?
> >> >
> >> > Thanks,
> >> > Sheryl
> >>
> >>
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Chris Mattmann, Ph.D.
> >> Senior Computer Scientist
> >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >> Office: 171-266B, Mailstop: 171-246
> >> Email: chris.a.mattmann@nasagov
> >>
> >> WWW: http://sunset.usc.edu/~mattmann/
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >> Adjunct Assistant Professor, Computer Science Department
> >> University of Southern California, Los Angeles, CA 90089 USA
> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >>
> >>
> >>
> >> --
> >> -Sheryl
> >
> >
> >
> > --
> > -Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Sheryl,
Are you sure that RecordID is being recorded for the particular products that you are trying to look up?
One way to find this out:
1. get the product IDs back for the products you are interested in. One way to do this would be
to remove RecordID from your query as you mention below and then add CAS.ProductId to the
SELECT list, and to change FORMAT='$FileLocation...' to FORMAT='$CAS.ProductId'.
2. Then, for each of the product IDs, check out OODT-306 [1] and use the metdump alias to dump
out the metadata for each product ID to the current working directory. Then, inspect that .met file.
Is RecordID recorded?
Let's start there.
Cheers,
Chris
[1] http://issues.apache.org/jira/browse/OODT-306
On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
> Hi,
>
> I am not clear about the metadata in .met files generated by the met-config.xml files. Particularly, where do they fall in the product-type and element mappings?
>
> Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are mapped to the product-type, ISMRawData and elements that are mapped to the GenericFile product-type.
>
> Except RecordID, which was defined as a key in the metout-config file and I'm assuming that the parsing error is because the RecordID is not mapped to any product-type. This query works and pulls all the files I need when I remove RecordID from the query. Again, since I'm not clear about metadata mapping , it's highly likely that I'm wrong about this.
>
> Thanks,
> On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> hey Sheryl,
>
> I think there is a misunderstanding on how the met-config.xml files are used... the metadata in these files are not added to your workflow context metadata... they are only used to create the *.met files for the data file for filemgr ingest... about your query I'm not quite clear on how you are wanting go use it... it kinda seems like you want to dynamically replace out RecordIO... in which case you're gonna need it to look more like RecordIO = '[PID]'
>
> -brian
>
>
> On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
>
>> Hi Chris,
>>
>> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
>>
>> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
>> I guess I have revise my workflow logic and metadata use before getting back to this query.
>>
>> Thanks!
>>
>>
>>
>> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> Hi Sheryl,
>>
>> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
>>
>> Try using single quotes on both and see if that fixes it.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>>
>> > Hi,
>> >
>> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
>> >
>> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
>> >
>> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
>> >
>> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
>> >
>> > Thanks,
>> > Sheryl
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasagov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> --
>> -Sheryl
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Sheryl,
Are you sure that RecordID is being recorded for the particular products that you are trying to look up?
One way to find this out:
1. get the product IDs back for the products you are interested in. One way to do this would be
to remove RecordID from your query as you mention below and then add CAS.ProductId to the
SELECT list, and to change FORMAT='$FileLocation...' to FORMAT='$CAS.ProductId'.
2. Then, for each of the product IDs, check out OODT-306 [1] and use the metdump alias to dump
out the metadata for each product ID to the current working directory. Then, inspect that .met file.
Is RecordID recorded?
Let's start there.
Cheers,
Chris
[1] http://issues.apache.org/jira/browse/OODT-306
On Sep 15, 2011, at 8:53 PM, Sheryl John wrote:
> Hi,
>
> I am not clear about the metadata in .met files generated by the met-config.xml files. Particularly, where do they fall in the product-type and element mappings?
>
> Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = '[PID]' , I'm selecting elements that are mapped to the product-type, ISMRawData and elements that are mapped to the GenericFile product-type.
>
> Except RecordID, which was defined as a key in the metout-config file and I'm assuming that the parsing error is because the RecordID is not mapped to any product-type. This query works and pulls all the files I need when I remove RecordID from the query. Again, since I'm not clear about metadata mapping , it's highly likely that I'm wrong about this.
>
> Thanks,
> On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> hey Sheryl,
>
> I think there is a misunderstanding on how the met-config.xml files are used... the metadata in these files are not added to your workflow context metadata... they are only used to create the *.met files for the data file for filemgr ingest... about your query I'm not quite clear on how you are wanting go use it... it kinda seems like you want to dynamically replace out RecordIO... in which case you're gonna need it to look more like RecordIO = '[PID]'
>
> -brian
>
>
> On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
>
>> Hi Chris,
>>
>> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
>>
>> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
>> I guess I have revise my workflow logic and metadata use before getting back to this query.
>>
>> Thanks!
>>
>>
>>
>> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> Hi Sheryl,
>>
>> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
>>
>> Try using single quotes on both and see if that fixes it.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>>
>> > Hi,
>> >
>> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
>> >
>> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
>> >
>> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
>> >
>> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
>> >
>> > Thanks,
>> > Sheryl
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasagov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>>
>>
>> --
>> -Sheryl
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
Hi,
I am not clear about the metadata in .met files generated by the
met-config.xml files. Particularly, where do they fall in the product-type
and element mappings?
Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT
FileLocation,Filename,ISMTable,*RecordID* FROM ISMRawData WHERE ISMTable =
'Chartevents' AND *RecordID* = '[PID]' , I'm selecting elements that are
mapped to the product-type, ISMRawData and elements that are mapped to the
GenericFile product-type.
Except *RecordID, *which was defined as a key in the metout-config file and
I'm assuming that the parsing error is because the RecordID is not mapped to
any product-type. This query works and pulls all the files I need when I
remove RecordID from the query. Again, since I'm not clear about metadata
mapping , it's highly likely that I'm wrong about this.
Thanks,
On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> hey Sheryl,
>
> I think there is a misunderstanding on how the met-config.xml files are
> used... the metadata in these files are not added to your workflow context
> metadata... they are only used to create the *.met files for the data file
> for filemgr ingest... about your query I'm not quite clear on how you are
> wanting go use it... it kinda seems like you want to dynamically replace out
> RecordIO... in which case you're gonna need it to look more like RecordIO =
> '[PID]'
>
> -brian
>
>
> On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for pointing that out. I was working on a different query and
> similar to the one I described. Though I've used only single quotes now, I'm
> still getting the same parsing error.
>
> But, I think what I intended to do with context metadata ( for my workflow)
> doesn't make sense for a single file product and rather, would apply for a
> group of files.
> I guess I have revise my workflow logic and metadata use before getting
> back to this query.
>
> Thanks!
>
>
>
> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sheryl,
>>
>> I notice you're using double quotes on the RecordID = "PID", yet you are
>> using single quotes on ISMTable = 'Chartevents'.
>>
>> Try using single quotes on both and see if that fixes it.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>>
>> > Hi,
>> >
>> > I have defined some key-val pairs for a file( say Output.csv) in a
>> metout-config.xml for my PGETask Workflow. However, after executing the
>> workflow, the met-config.xml is not creating a Output.csv.cas file.
>> >
>> > I want to be able to use the above keys/metadata later on in an SQL-like
>> query from the pgeconfig file. For example, if I've defined 'RecordID' as a
>> key in the metout-config.xml, I would want to use this metadata in the
>> following query:
>> >
>> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
>> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
>> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
>> above are elements and product-types that were defined during ingestion in
>> the File Manager.
>> >
>> > At present, the task fails to parse the above query when I run the
>> workflow. Is this because metout-config is not creating the Output.cas file?
>> And, what is the best way to specify metadata files for a group of files or
>> for a folder?
>> >
>> > Thanks,
>> > Sheryl
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasagov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>
>
> --
> -Sheryl
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
Hi,
I am not clear about the metadata in .met files generated by the
met-config.xml files. Particularly, where do they fall in the product-type
and element mappings?
Because in my query, SQL(FORMAT='$FileLocation/$Filename'){ SELECT
FileLocation,Filename,ISMTable,*RecordID* FROM ISMRawData WHERE ISMTable =
'Chartevents' AND *RecordID* = '[PID]' , I'm selecting elements that are
mapped to the product-type, ISMRawData and elements that are mapped to the
GenericFile product-type.
Except *RecordID, *which was defined as a key in the metout-config file and
I'm assuming that the parsing error is because the RecordID is not mapped to
any product-type. This query works and pulls all the files I need when I
remove RecordID from the query. Again, since I'm not clear about metadata
mapping , it's highly likely that I'm wrong about this.
Thanks,
On Thu, Sep 15, 2011 at 11:17 AM, holenoter <ho...@mac.com> wrote:
> hey Sheryl,
>
> I think there is a misunderstanding on how the met-config.xml files are
> used... the metadata in these files are not added to your workflow context
> metadata... they are only used to create the *.met files for the data file
> for filemgr ingest... about your query I'm not quite clear on how you are
> wanting go use it... it kinda seems like you want to dynamically replace out
> RecordIO... in which case you're gonna need it to look more like RecordIO =
> '[PID]'
>
> -brian
>
>
> On Sep 14, 2011, at 09:18 PM, Sheryl John <sh...@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for pointing that out. I was working on a different query and
> similar to the one I described. Though I've used only single quotes now, I'm
> still getting the same parsing error.
>
> But, I think what I intended to do with context metadata ( for my workflow)
> doesn't make sense for a single file product and rather, would apply for a
> group of files.
> I guess I have revise my workflow logic and metadata use before getting
> back to this query.
>
> Thanks!
>
>
>
> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> Hi Sheryl,
>>
>> I notice you're using double quotes on the RecordID = "PID", yet you are
>> using single quotes on ISMTable = 'Chartevents'.
>>
>> Try using single quotes on both and see if that fixes it.
>>
>> Thanks!
>>
>> Cheers,
>> Chris
>>
>>
>> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>>
>> > Hi,
>> >
>> > I have defined some key-val pairs for a file( say Output.csv) in a
>> metout-config.xml for my PGETask Workflow. However, after executing the
>> workflow, the met-config.xml is not creating a Output.csv.cas file.
>> >
>> > I want to be able to use the above keys/metadata later on in an SQL-like
>> query from the pgeconfig file. For example, if I've defined 'RecordID' as a
>> key in the metout-config.xml, I would want to use this metadata in the
>> following query:
>> >
>> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
>> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
>> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
>> above are elements and product-types that were defined during ingestion in
>> the File Manager.
>> >
>> > At present, the task fails to parse the above query when I run the
>> workflow. Is this because metout-config is not creating the Output.cas file?
>> And, what is the best way to specify metadata files for a group of files or
>> for a folder?
>> >
>> > Thanks,
>> > Sheryl
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasagov
>>
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>>
>
>
> --
> -Sheryl
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Okey dok, Sheryl keep us posted!
Cheers,
Chris
On Sep 14, 2011, at 9:18 PM, Sheryl John wrote:
> Hi Chris,
>
> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
>
> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
> I guess I have revise my workflow logic and metadata use before getting back to this query.
>
> Thanks!
>
>
>
> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> Hi Sheryl,
>
> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
>
> Try using single quotes on both and see if that fixes it.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
> >
> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
> >
> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
> >
> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
> >
> > Thanks,
> > Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Okey dok, Sheryl keep us posted!
Cheers,
Chris
On Sep 14, 2011, at 9:18 PM, Sheryl John wrote:
> Hi Chris,
>
> Thanks for pointing that out. I was working on a different query and similar to the one I described. Though I've used only single quotes now, I'm still getting the same parsing error.
>
> But, I think what I intended to do with context metadata ( for my workflow) doesn't make sense for a single file product and rather, would apply for a group of files.
> I guess I have revise my workflow logic and metadata use before getting back to this query.
>
> Thanks!
>
>
>
> On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> Hi Sheryl,
>
> I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
>
> Try using single quotes on both and see if that fixes it.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
> >
> > I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
> >
> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
> >
> > At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
> >
> > Thanks,
> > Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
>
>
> --
> -Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
Hi Chris,
Thanks for pointing that out. I was working on a different query and similar
to the one I described. Though I've used only single quotes now, I'm still
getting the same parsing error.
But, I think what I intended to do with context metadata ( for my workflow)
doesn't make sense for a single file product and rather, would apply for a
group of files.
I guess I have revise my workflow logic and metadata use before getting back
to this query.
Thanks!
On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sheryl,
>
> I notice you're using double quotes on the RecordID = "PID", yet you are
> using single quotes on ISMTable = 'Chartevents'.
>
> Try using single quotes on both and see if that fixes it.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I have defined some key-val pairs for a file( say Output.csv) in a
> metout-config.xml for my PGETask Workflow. However, after executing the
> workflow, the met-config.xml is not creating a Output.csv.cas file.
> >
> > I want to be able to use the above keys/metadata later on in an SQL-like
> query from the pgeconfig file. For example, if I've defined 'RecordID' as a
> key in the metout-config.xml, I would want to use this metadata in the
> following query:
> >
> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
> above are elements and product-types that were defined during ingestion in
> the File Manager.
> >
> > At present, the task fails to parse the above query when I run the
> workflow. Is this because metout-config is not creating the Output.cas file?
> And, what is the best way to specify metadata files for a group of files or
> for a folder?
> >
> > Thanks,
> > Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by Sheryl John <sh...@gmail.com>.
Hi Chris,
Thanks for pointing that out. I was working on a different query and similar
to the one I described. Though I've used only single quotes now, I'm still
getting the same parsing error.
But, I think what I intended to do with context metadata ( for my workflow)
doesn't make sense for a single file product and rather, would apply for a
group of files.
I guess I have revise my workflow logic and metadata use before getting back
to this query.
Thanks!
On Wed, Sep 14, 2011 at 8:42 PM, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Sheryl,
>
> I notice you're using double quotes on the RecordID = "PID", yet you are
> using single quotes on ISMTable = 'Chartevents'.
>
> Try using single quotes on both and see if that fixes it.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
>
> > Hi,
> >
> > I have defined some key-val pairs for a file( say Output.csv) in a
> metout-config.xml for my PGETask Workflow. However, after executing the
> workflow, the met-config.xml is not creating a Output.csv.cas file.
> >
> > I want to be able to use the above keys/metadata later on in an SQL-like
> query from the pgeconfig file. For example, if I've defined 'RecordID' as a
> key in the metout-config.xml, I would want to use this metadata in the
> following query:
> >
> > SQL(FORMAT='$FileLocation/$Filename'){ SELECT
> FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable =
> 'Chartevents' AND RecordID = "PID"} The others keys included in the query
> above are elements and product-types that were defined during ingestion in
> the File Manager.
> >
> > At present, the task fails to parse the above query when I run the
> workflow. Is this because metout-config is not creating the Output.cas file?
> And, what is the best way to specify metadata files for a group of files or
> for a folder?
> >
> > Thanks,
> > Sheryl
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
--
-Sheryl
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sheryl,
I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
Try using single quotes on both and see if that fixes it.
Thanks!
Cheers,
Chris
On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> Hi,
>
> I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
>
> I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
>
> SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
>
> At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
>
> Thanks,
> Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: PGETask Metadata
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Sheryl,
I notice you're using double quotes on the RecordID = "PID", yet you are using single quotes on ISMTable = 'Chartevents'.
Try using single quotes on both and see if that fixes it.
Thanks!
Cheers,
Chris
On Sep 14, 2011, at 7:03 PM, Sheryl John wrote:
> Hi,
>
> I have defined some key-val pairs for a file( say Output.csv) in a metout-config.xml for my PGETask Workflow. However, after executing the workflow, the met-config.xml is not creating a Output.csv.cas file.
>
> I want to be able to use the above keys/metadata later on in an SQL-like query from the pgeconfig file. For example, if I've defined 'RecordID' as a key in the metout-config.xml, I would want to use this metadata in the following query:
>
> SQL(FORMAT='$FileLocation/$Filename'){ SELECT FileLocation,Filename,ISMTable,RecordID FROM ISMRawData WHERE ISMTable = 'Chartevents' AND RecordID = "PID"} The others keys included in the query above are elements and product-types that were defined during ingestion in the File Manager.
>
> At present, the task fails to parse the above query when I run the workflow. Is this because metout-config is not creating the Output.cas file? And, what is the best way to specify metadata files for a group of files or for a folder?
>
> Thanks,
> Sheryl
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++