You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Thomas Bennett <tb...@ska.ac.za> on 2011/10/19 15:22:33 UTC
query syntax
Hi,
I'm trying out wildcards when using query_tool to run some queries on a
lucene catalog and its throwing exceptions left right and center.
Any help/pointers welcome.
This query:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT Filename
FROM KatFile WHERE Observer='ja*per'"
Returns:
Oct 19, 2011 3:19:53 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog
paginateQuery
WARNING: Query: [q=Observer:ja*per] for Product Type: [urn:kat:KatFile]
returned no results
java.lang.NullPointerException
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFileManager.java:602)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:680)
org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to
perform complex query : null
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at
org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at
org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main"
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
java.lang.Exception:
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to
perform complex query : null
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at
org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)--
Thomas Bennett
SKA South Africa
Office : +2721 506 7341
Mobile : +2779 523 7105
Email : tbennett@ska.ac.za
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
+1, sounds like a great approach.
Cheers,
Chris
On Nov 1, 2011, at 3:04 AM, Thomas Bennett wrote:
> Hi,
>
> Chris - thanks again for your reply and our Skype conversation.
>
> To complete this email thread - it would seam that the query syntax allowed by the File Manager is restricted and that the suggested approach is to use the File Manager Query for doing searches from system level services (and other components in OODT).
>
> For user queries it would be better to have a nightly dump of the catalogue and use Solr.
>
> Cheers,
> Tom
>
> --
> Thomas Bennett
>
> SKA South Africa
>
> Office : +2721 506 7341
> Mobile : +2779 523 7105
> Email : tbennett@ska.ac.za
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
+1, sounds like a great approach.
Cheers,
Chris
On Nov 1, 2011, at 3:04 AM, Thomas Bennett wrote:
> Hi,
>
> Chris - thanks again for your reply and our Skype conversation.
>
> To complete this email thread - it would seam that the query syntax allowed by the File Manager is restricted and that the suggested approach is to use the File Manager Query for doing searches from system level services (and other components in OODT).
>
> For user queries it would be better to have a nightly dump of the catalogue and use Solr.
>
> Cheers,
> Tom
>
> --
> Thomas Bennett
>
> SKA South Africa
>
> Office : +2721 506 7341
> Mobile : +2779 523 7105
> Email : tbennett@ska.ac.za
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <tb...@ska.ac.za>.
Hi,
Chris - thanks again for your reply and our Skype conversation.
To complete this email thread - it would seam that the query syntax allowed
by the File Manager is restricted and that the suggested approach is to use
the File Manager Query for doing searches from system level services (and
other components in OODT).
For user queries it would be better to have a nightly dump of the catalogue
and use Solr.
Cheers,
Tom
--
Thomas Bennett
SKA South Africa
Office : +2721 506 7341
Mobile : +2779 523 7105
Email : tbennett@ska.ac.za
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
No problem. As you saw, Paul has been helping me out.
Two day meetings... eish. Hope you survived!
Cheers,
Tom
On 13 April 2012 08:48, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Tom,
>
> I'm just now getting to this, sorry been in meetings all day the past 2
> days (Open Source Think
> Tank).
>
> Anyways, done now, and looks like you've sorted this out -- it was because
> UUID was a legacy
> field in the project that originally used this code -- however it's really
> CAS.ProductId, and glad
> we are pushing for that.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Apr 11, 2012, at 6:05 AM, Thomas Bennett wrote:
>
> > Hey Chris,
> >
> > Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
> >
> > SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
> >
> > Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
> >
> > Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
> >
> > map.ExperimentID=id
> > map.Observer=author
> > map.Description=description
> >
> > Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
> >
> > <!-- Field to use to determine and enforce document uniqueness.
> > Unless this field is marked with required="false", it will be a
> required field
> > -->
> > <uniqueKey>id</uniqueKey>
> >
> > Okay, so I finally ran the SolrIndexer with the -all.
> >
> > This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
> >
> > if (metadata.getMetadata("UUID") != null) {
> > if (metadata.getMetadata("Deleted") == null
> >
> > || !"true".equals(metadata.getMetadata("Deleted"))) {
> >
> > try {
> >
> > server.add(this.getSolrDocument(metadata));
> >
> > server.commit();
> >
> > LOG.info("Indexed " + metadata.getMetadata("UUID"));
> >
> > } catch (Exception e) {
> >
> > LOG.severe("Could not index " +
> metadata.getMetadata("UUID")
> >
> > + " " + e.getMessage());
> >
> > }
> >
> > } else {
> >
> > LOG.info("Skipping Deleted: " +
> metadata.getMetadata("UUID"));
> >
> > }
> >
> > }
> >
> > Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
> >
> > I've tried the single product index, which works just fine.
> >
> > Cheers,
> > Tom
> >
> > On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
> > No worries...based on your email below, did you update the Solr schema?
> I think you need
> > to tell it to allow for all fields, by uncommenting the dynamicField
> block in the schema.xml file
> > to say "allow all fields" to come through, or modify the schema.xml in
> Solr's conf directory to
> > specify which fields to send to Solr.
> >
> > Does that make sense? I can provide more detailed help and will follow
> up later, but
> > see if that helps for now.
> >
> > Cheers,
> > Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
No problem. As you saw, Paul has been helping me out.
Two day meetings... eish. Hope you survived!
Cheers,
Tom
On 13 April 2012 08:48, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> Hey Tom,
>
> I'm just now getting to this, sorry been in meetings all day the past 2
> days (Open Source Think
> Tank).
>
> Anyways, done now, and looks like you've sorted this out -- it was because
> UUID was a legacy
> field in the project that originally used this code -- however it's really
> CAS.ProductId, and glad
> we are pushing for that.
>
> Thanks!
>
> Cheers,
> Chris
>
> On Apr 11, 2012, at 6:05 AM, Thomas Bennett wrote:
>
> > Hey Chris,
> >
> > Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
> >
> > SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
> >
> > Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
> >
> > Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
> >
> > map.ExperimentID=id
> > map.Observer=author
> > map.Description=description
> >
> > Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
> >
> > <!-- Field to use to determine and enforce document uniqueness.
> > Unless this field is marked with required="false", it will be a
> required field
> > -->
> > <uniqueKey>id</uniqueKey>
> >
> > Okay, so I finally ran the SolrIndexer with the -all.
> >
> > This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
> >
> > if (metadata.getMetadata("UUID") != null) {
> > if (metadata.getMetadata("Deleted") == null
> >
> > || !"true".equals(metadata.getMetadata("Deleted"))) {
> >
> > try {
> >
> > server.add(this.getSolrDocument(metadata));
> >
> > server.commit();
> >
> > LOG.info("Indexed " + metadata.getMetadata("UUID"));
> >
> > } catch (Exception e) {
> >
> > LOG.severe("Could not index " +
> metadata.getMetadata("UUID")
> >
> > + " " + e.getMessage());
> >
> > }
> >
> > } else {
> >
> > LOG.info("Skipping Deleted: " +
> metadata.getMetadata("UUID"));
> >
> > }
> >
> > }
> >
> > Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
> >
> > I've tried the single product index, which works just fine.
> >
> > Cheers,
> > Tom
> >
> > On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
> > No worries...based on your email below, did you update the Solr schema?
> I think you need
> > to tell it to allow for all fields, by uncommenting the dynamicField
> block in the schema.xml file
> > to say "allow all fields" to come through, or modify the schema.xml in
> Solr's conf directory to
> > specify which fields to send to Solr.
> >
> > Does that make sense? I can provide more detailed help and will follow
> up later, but
> > see if that helps for now.
> >
> > Cheers,
> > Chris
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
I'm just now getting to this, sorry been in meetings all day the past 2 days (Open Source Think
Tank).
Anyways, done now, and looks like you've sorted this out -- it was because UUID was a legacy
field in the project that originally used this code -- however it's really CAS.ProductId, and glad
we are pushing for that.
Thanks!
Cheers,
Chris
On Apr 11, 2012, at 6:05 AM, Thomas Bennett wrote:
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>
> if (metadata.getMetadata("UUID") != null) {
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>
> }
>
> }
>
> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up later, but
> see if that helps for now.
>
> Cheers,
> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Paul,
Thanks for the reply!
That explains it then. Useful tool - I'm glad this made its way into the
code base :).
I'll create a JIRA issue and patch.
Cheers,
Tom
On 12 April 2012 12:57, Ramirez, Paul M (388J)
<pa...@jpl.nasa.gov>wrote:
> Hey Tom,
>
> To follow up on this lets enter a Jira issue and the patch should be to
> that outer " if (metadata.getMetadata("UUID") != null) {" and then
> change the logging messages there to have
> metadata.getMetadata("CAS.ProductId"). In addition, the else part of that
> statement can be dropped.
>
> I also found one more place where the uuid appears and that is in the
> indeMetFile method. Here this should be updated to
> metadata.getMetadata("CAS.ProductId").
>
> Thanks,
> Paul
>
> From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
> Date: Thu, 12 Apr 2012 03:32:27 -0700
> To: "<us...@oodt.apache.org>" <us...@oodt.apache.org>
> Cc: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: query syntax
>
> This UUID stuff should have been dropped when it was ported to Apache.
> This was a project specific thing where the team that developed it had a
> different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>
> wrote:
>
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a
> required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
>
> if (metadata.getMetadata("UUID") != null) {
>
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata(
> "UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata(
> "UUID"));
>
> }
>
> }
> Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> No worries...based on your email below, did you update the Solr schema? I
>> think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField
>> block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in
>> Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up
>> later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Paul,
Thanks for the reply!
That explains it then. Useful tool - I'm glad this made its way into the
code base :).
I'll create a JIRA issue and patch.
Cheers,
Tom
On 12 April 2012 12:57, Ramirez, Paul M (388J)
<pa...@jpl.nasa.gov>wrote:
> Hey Tom,
>
> To follow up on this lets enter a Jira issue and the patch should be to
> that outer " if (metadata.getMetadata("UUID") != null) {" and then
> change the logging messages there to have
> metadata.getMetadata("CAS.ProductId"). In addition, the else part of that
> statement can be dropped.
>
> I also found one more place where the uuid appears and that is in the
> indeMetFile method. Here this should be updated to
> metadata.getMetadata("CAS.ProductId").
>
> Thanks,
> Paul
>
> From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
> Date: Thu, 12 Apr 2012 03:32:27 -0700
> To: "<us...@oodt.apache.org>" <us...@oodt.apache.org>
> Cc: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: query syntax
>
> This UUID stuff should have been dropped when it was ported to Apache.
> This was a project specific thing where the team that developed it had a
> different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>
> wrote:
>
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a
> required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
>
> if (metadata.getMetadata("UUID") != null) {
>
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata(
> "UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata(
> "UUID"));
>
> }
>
> }
> Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> No worries...based on your email below, did you update the Solr schema? I
>> think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField
>> block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in
>> Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up
>> later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
Thanks. This has to do with namespacing and the need to support
legacy users of the software wherein which CAS.ProductId was
the standard field used. To support this, in the CoreMetExtractor FM
met extractor, we allow namespacing (e.g., "CAS." to be used on the core met keys).
However, newer users of the system have a path for not having to use CAS.
if needed. It's not recommended though since I am not sure of the full
widespread use (e.g., in downstream UIs, etc.) of CAS as a prefix.
That being said, I think CoreMetKeys.PRODUCT_ID is *still* correct,
as "ProductId", however, in this case, I put (on my comment on r/4709/
that we should do: "CAS." + CoreMetKeys.PRODUCT_ID.
Sound good? +1 for that.
Cheers,
Chris
On Apr 13, 2012, at 5:07 AM, Thomas Bennett wrote:
> Hey Paul and Chris,
>
> I'm just following up with an email - see my comment in https://reviews.apache.org/r/4709/.
>
> It seems that CoreMetKeys.PRODUCT_ID = "ProductId" and not "CAS.ProductId" as specified in the cas-filemgr/policy/elements.xml file.
>
> Should they be the same?
>
> Cheers,
> Tom
>
> On 12 April 2012 12:32, Ramirez, Paul M (388J) <pa...@jpl.nasa.gov> wrote:
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>>
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>>
>> try {
>>
>> server.add(this.getSolrDocument(metadata));
>>
>> server.commit();
>>
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>>
>> } catch (Exception e) {
>>
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>>
>> + " " + e.getMessage());
>>
>> }
>>
>> } else {
>>
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>>
>> }
>>
>> }
>>
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> No worries...based on your email below, did you update the Solr schema? I think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
Thanks. This has to do with namespacing and the need to support
legacy users of the software wherein which CAS.ProductId was
the standard field used. To support this, in the CoreMetExtractor FM
met extractor, we allow namespacing (e.g., "CAS." to be used on the core met keys).
However, newer users of the system have a path for not having to use CAS.
if needed. It's not recommended though since I am not sure of the full
widespread use (e.g., in downstream UIs, etc.) of CAS as a prefix.
That being said, I think CoreMetKeys.PRODUCT_ID is *still* correct,
as "ProductId", however, in this case, I put (on my comment on r/4709/
that we should do: "CAS." + CoreMetKeys.PRODUCT_ID.
Sound good? +1 for that.
Cheers,
Chris
On Apr 13, 2012, at 5:07 AM, Thomas Bennett wrote:
> Hey Paul and Chris,
>
> I'm just following up with an email - see my comment in https://reviews.apache.org/r/4709/.
>
> It seems that CoreMetKeys.PRODUCT_ID = "ProductId" and not "CAS.ProductId" as specified in the cas-filemgr/policy/elements.xml file.
>
> Should they be the same?
>
> Cheers,
> Tom
>
> On 12 April 2012 12:32, Ramirez, Paul M (388J) <pa...@jpl.nasa.gov> wrote:
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>>
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>>
>> try {
>>
>> server.add(this.getSolrDocument(metadata));
>>
>> server.commit();
>>
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>>
>> } catch (Exception e) {
>>
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>>
>> + " " + e.getMessage());
>>
>> }
>>
>> } else {
>>
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>>
>> }
>>
>> }
>>
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> No worries...based on your email below, did you update the Solr schema? I think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Paul and Chris,
I'm just following up with an email - see my comment in
https://reviews.apache.org/r/4709/.
It seems that CoreMetKeys.PRODUCT_ID = "ProductId" and not "CAS.ProductId"
as specified in the cas-filemgr/policy/elements.xml file.
Should they be the same?
Cheers,
Tom
On 12 April 2012 12:32, Ramirez, Paul M (388J)
<pa...@jpl.nasa.gov>wrote:
> This UUID stuff should have been dropped when it was ported to Apache.
> This was a project specific thing where the team that developed it had a
> different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>
> wrote:
>
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a
> required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
>
> if (metadata.getMetadata("UUID") != null) {
>
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata(
> "UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata(
> "UUID"));
>
> }
>
> }
> Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> No worries...based on your email below, did you update the Solr schema? I
>> think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField
>> block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in
>> Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up
>> later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Yeah my bad, I should have caught this when porting it over, but didn't!
Cheers,
Chris
On Apr 12, 2012, at 3:32 AM, Ramirez, Paul M (388J) wrote:
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>>
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>>
>> try {
>>
>> server.add(this.getSolrDocument(metadata));
>>
>> server.commit();
>>
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>>
>> } catch (Exception e) {
>>
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>>
>> + " " + e.getMessage());
>>
>> }
>>
>> } else {
>>
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>>
>> }
>>
>> }
>>
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> No worries...based on your email below, did you update the Solr schema? I think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Paul and Chris,
I'm just following up with an email - see my comment in
https://reviews.apache.org/r/4709/.
It seems that CoreMetKeys.PRODUCT_ID = "ProductId" and not "CAS.ProductId"
as specified in the cas-filemgr/policy/elements.xml file.
Should they be the same?
Cheers,
Tom
On 12 April 2012 12:32, Ramirez, Paul M (388J)
<pa...@jpl.nasa.gov>wrote:
> This UUID stuff should have been dropped when it was ported to Apache.
> This was a project specific thing where the team that developed it had a
> different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>
> wrote:
>
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my
> metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a
> File Manager and put them into a Solr index. It does this by an http POST
> to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the
> actual POST and see what the content is :). This is how I discovered that
> the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these
> fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the
> solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a
> required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never
> executed. It's checking for metadata "UUID". I don't have this metadata
> key.
>
> if (metadata.getMetadata("UUID") != null) {
>
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata(
> "UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata(
> "UUID"));
>
> }
>
> }
> Question: Deleted looks optional, so no problem there, but what should I
> be doing with UUID? I've changed this to be CAS.ProductId in the code, just
> to check nothing else was causing a problem. I thought it would be worth
> asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
>> No worries...based on your email below, did you update the Solr schema? I
>> think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField
>> block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in
>> Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up
>> later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
>
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Yeah my bad, I should have caught this when porting it over, but didn't!
Cheers,
Chris
On Apr 12, 2012, at 3:32 AM, Ramirez, Paul M (388J) wrote:
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>>
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>>
>> try {
>>
>> server.add(this.getSolrDocument(metadata));
>>
>> server.commit();
>>
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>>
>> } catch (Exception e) {
>>
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>>
>> + " " + e.getMessage());
>>
>> }
>>
>> } else {
>>
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>>
>> }
>>
>> }
>>
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>> No worries...based on your email below, did you update the Solr schema? I think you need
>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>> specify which fields to send to Solr.
>>
>> Does that make sense? I can provide more detailed help and will follow up later, but
>> see if that helps for now.
>>
>> Cheers,
>> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1...
Cheers,
Chris
On Apr 12, 2012, at 3:57 AM, Ramirez, Paul M (388J) wrote:
> Hey Tom,
>
> To follow up on this lets enter a Jira issue and the patch should be to that outer " if (metadata.getMetadata("UUID") != null) {" and then change the logging messages there to have metadata.getMetadata("CAS.ProductId"). In addition, the else part of that statement can be dropped.
>
> I also found one more place where the uuid appears and that is in the indeMetFile method. Here this should be updated to metadata.getMetadata("CAS.ProductId").
>
> Thanks,
> Paul
>
> From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
> Date: Thu, 12 Apr 2012 03:32:27 -0700
> To: "<us...@oodt.apache.org>" <us...@oodt.apache.org>
> Cc: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: query syntax
>
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>> try {
>> server.add(this.getSolrDocument(metadata));
>> server.commit();
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>> } catch (Exception e) {
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>> + " " + e.getMessage());
>> }
>> } else {
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>> }
>> }
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>>> No worries...based on your email below, did you update the Solr schema? I think you need
>>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>>> specify which fields to send to Solr.
>>>
>>> Does that make sense? I can provide more detailed help and will follow up later, but
>>> see if that helps for now.
>>>
>>> Cheers,
>>> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1...
Cheers,
Chris
On Apr 12, 2012, at 3:57 AM, Ramirez, Paul M (388J) wrote:
> Hey Tom,
>
> To follow up on this lets enter a Jira issue and the patch should be to that outer " if (metadata.getMetadata("UUID") != null) {" and then change the logging messages there to have metadata.getMetadata("CAS.ProductId"). In addition, the else part of that statement can be dropped.
>
> I also found one more place where the uuid appears and that is in the indeMetFile method. Here this should be updated to metadata.getMetadata("CAS.ProductId").
>
> Thanks,
> Paul
>
> From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>
> Date: Thu, 12 Apr 2012 03:32:27 -0700
> To: "<us...@oodt.apache.org>" <us...@oodt.apache.org>
> Cc: "user@oodt.apache.org" <us...@oodt.apache.org>
> Subject: Re: query syntax
>
> This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
>
> --Paul
>
> On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com> wrote:
>
>> Hey Chris,
>>
>> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>>
>> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>>
>> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>>
>> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>>
>> map.ExperimentID=id
>> map.Observer=author
>> map.Description=description
>>
>> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>>
>> <!-- Field to use to determine and enforce document uniqueness.
>> Unless this field is marked with required="false", it will be a required field
>> -->
>> <uniqueKey>id</uniqueKey>
>>
>> Okay, so I finally ran the SolrIndexer with the -all.
>>
>> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>>
>> if (metadata.getMetadata("UUID") != null) {
>> if (metadata.getMetadata("Deleted") == null
>> || !"true".equals(metadata.getMetadata("Deleted"))) {
>> try {
>> server.add(this.getSolrDocument(metadata));
>> server.commit();
>> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>> } catch (Exception e) {
>> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>> + " " + e.getMessage());
>> }
>> } else {
>> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>> }
>> }
>> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>>
>> I've tried the single product index, which works just fine.
>>
>> Cheers,
>> Tom
>>
>> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
>>> No worries...based on your email below, did you update the Solr schema? I think you need
>>> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
>>> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
>>> specify which fields to send to Solr.
>>>
>>> Does that make sense? I can provide more detailed help and will follow up later, but
>>> see if that helps for now.
>>>
>>> Cheers,
>>> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>.
Hey Tom,
To follow up on this lets enter a Jira issue and the patch should be to that outer " if (metadata.getMetadata("UUID") != null) {" and then change the logging messages there to have metadata.getMetadata("CAS.ProductId"). In addition, the else part of that statement can be dropped.
I also found one more place where the uuid appears and that is in the indeMetFile method. Here this should be updated to metadata.getMetadata("CAS.ProductId").
Thanks,
Paul
From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Date: Thu, 12 Apr 2012 03:32:27 -0700
To: "<us...@oodt.apache.org>>" <us...@oodt.apache.org>>
Cc: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: query syntax
This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
--Paul
On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata("UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
}
}
Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>> wrote:
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
Re: query syntax
Posted by "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>.
Hey Tom,
To follow up on this lets enter a Jira issue and the patch should be to that outer " if (metadata.getMetadata("UUID") != null) {" and then change the logging messages there to have metadata.getMetadata("CAS.ProductId"). In addition, the else part of that statement can be dropped.
I also found one more place where the uuid appears and that is in the indeMetFile method. Here this should be updated to metadata.getMetadata("CAS.ProductId").
Thanks,
Paul
From: "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>>
Date: Thu, 12 Apr 2012 03:32:27 -0700
To: "<us...@oodt.apache.org>>" <us...@oodt.apache.org>>
Cc: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: query syntax
This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
--Paul
On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata("UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
}
}
Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>> wrote:
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
Re: query syntax
Posted by "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>.
This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
--Paul
On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata("UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
}
}
Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>> wrote:
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
Re: query syntax
Posted by "Ramirez, Paul M (388J)" <pa...@jpl.nasa.gov>.
This UUID stuff should have been dropped when it was ported to Apache. This was a project specific thing where the team that developed it had a different unique id (UUID).
--Paul
On Apr 11, 2012, at 6:06 PM, "Thomas Bennett" <lm...@gmail.com>> wrote:
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata("UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
}
}
Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov>> wrote:
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
I'm just now getting to this, sorry been in meetings all day the past 2 days (Open Source Think
Tank).
Anyways, done now, and looks like you've sorted this out -- it was because UUID was a legacy
field in the project that originally used this code -- however it's really CAS.ProductId, and glad
we are pushing for that.
Thanks!
Cheers,
Chris
On Apr 11, 2012, at 6:05 AM, Thomas Bennett wrote:
> Hey Chris,
>
> Here's a basic summary of what I've discovered so far getting my metadata into a Solr index...
>
> SolrIndexer queries metadata for a single or multiple products from a File Manager and put them into a Solr index. It does this by an http POST to solr/update. Very cool.
>
> Because it's xml on the wire, its really easy to tcpdump and view the actual POST and see what the content is :). This is how I discovered that the indexAll was not updating solr (see below).
>
> Okay so for a simple test, I set up the following mapping, since these fields already exist in the Solr schema.xml file:
>
> map.ExperimentID=id
> map.Observer=author
> map.Description=description
>
> Note: It seems a good idea to have an "id" field. If you look in the solr scheme.xml file, I saw the following configuration by default:
>
> <!-- Field to use to determine and enforce document uniqueness.
> Unless this field is marked with required="false", it will be a required field
> -->
> <uniqueKey>id</uniqueKey>
>
> Okay, so I finally ran the SolrIndexer with the -all.
>
> This piece of code (starting at line 209 of SolrIndexer.java) was never executed. It's checking for metadata "UUID". I don't have this metadata key.
>
> if (metadata.getMetadata("UUID") != null) {
> if (metadata.getMetadata("Deleted") == null
>
> || !"true".equals(metadata.getMetadata("Deleted"))) {
>
> try {
>
> server.add(this.getSolrDocument(metadata));
>
> server.commit();
>
> LOG.info("Indexed " + metadata.getMetadata("UUID"));
>
> } catch (Exception e) {
>
> LOG.severe("Could not index " + metadata.getMetadata("UUID")
>
> + " " + e.getMessage());
>
> }
>
> } else {
>
> LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"));
>
> }
>
> }
>
> Question: Deleted looks optional, so no problem there, but what should I be doing with UUID? I've changed this to be CAS.ProductId in the code, just to check nothing else was causing a problem. I thought it would be worth asking why UUID is specified...
>
> I've tried the single product index, which works just fine.
>
> Cheers,
> Tom
>
> On 30 March 2012 20:38, Mattmann, Chris A (388J) <ch...@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up later, but
> see if that helps for now.
>
> Cheers,
> Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata
into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File
Manager and put them into a Solr index. It does this by an http POST to
solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the
actual POST and see what the content is :). This is how I discovered that
the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these
fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr
scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a
required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never
executed. It's checking for metadata "UUID". I don't have this metadata
key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata(
"UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"
));
}
}
Question: Deleted looks optional, so no problem there, but what should I be
doing with UUID? I've changed this to be CAS.ProductId in the code, just to
check nothing else was causing a problem. I thought it would be worth
asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I
> think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block
> in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in
> Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up
> later, but
> see if that helps for now.
>
> Cheers,
> Chris
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hi Sean,
Thanks for the info! I was unsure which of the Python API's to look into. I'll take a look :-).
Regards,
Tom.
Sent from my iPhone
On 30 Mar 2012, at 9:02 PM, Sean Kelly <ke...@apache.org> wrote:
>> So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
>
> Double woot indeed!
>
> And if you're not up to handling JSON, there's a Python API:
>
> https://github.com/toastdriven/pysolr/
>
> --k
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hi Sean,
Thanks for the info! I was unsure which of the Python API's to look into. I'll take a look :-).
Regards,
Tom.
Sent from my iPhone
On 30 Mar 2012, at 9:02 PM, Sean Kelly <ke...@apache.org> wrote:
>> So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
>
> Double woot indeed!
>
> And if you're not up to handling JSON, there's a Python API:
>
> https://github.com/toastdriven/pysolr/
>
> --k
>
Re: query syntax
Posted by Sean Kelly <ke...@apache.org>.
> So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
Double woot indeed!
And if you're not up to handling JSON, there's a Python API:
https://github.com/toastdriven/pysolr/
--k
Re: query syntax
Posted by Sean Kelly <ke...@apache.org>.
> So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
Double woot indeed!
And if you're not up to handling JSON, there's a Python API:
https://github.com/toastdriven/pysolr/
--k
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Thanks for the feedback. Makes perfect sense. I'll check it out tomorrow and let you know.
So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
Sent from my iPhone
On 30 Mar 2012, at 8:38 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up later, but
> see if that helps for now.
>
> Cheers,
> Chris
>
> On Mar 30, 2012, at 7:54 AM, Thomas Bennett wrote:
>
>> Hey Chris,
>>
>> Sorry for the vagueness of this email... I'm trying to get solr working, but I seem to be doing something wrong...
>>
>> Here's my indexer.properties file:
>>
>> # the URL path to Solr
>> config.solr.url=http://192.168.1.51:8983/solr
>>
>> # the URL path to the File Manager
>> config.filemgr.url=http://localhost:9101
>>
>> config.ref.DataDownloadRef=dataref
>>
>> # a set of products to map from File Manager
>> # terminology into Solr index doc field
>> # terminology
>> map.MimeType=mimetype
>> map.ReceivedTime=receivedtime
>> map.FileSize=filesize
>> map.FileName=filename
>> map.Antennas=antennas
>> map.Observer=observer
>> map.Description=description
>> map.StartTime=starttime
>>
>> I then ran:
>>
>> ~$ java -DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties -Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/ org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl http://localhost:9101
>>
>> Which returned things like:
>> ....
>> INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
>> 30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer indexAll
>> ...
>>
>> However when I tried to search from solr admin interface I tried a wild card search to find anything:
>>
>> *:*
>>
>> Also tried:
>>
>> observer:thomas which returned:
>> undefined field observer
>>
>> Looks like I'm not getting the index across properly.
>>
>> Any ideas....?
>>
>> Tom
>>
>> On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
>> Hey Chris,
>>
>> Thanks for the info :-) much obliged.
>>
>> Tom.
>>
>> Sent from my iPhone
>>
>> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Tom,
>>>
>>> No problem, here is the config sample:
>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
>>>
>>> HTH!
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>> On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
>>>
>>>> Hey Chris / Paul,
>>>>
>>>> Thanks for the info.
>>>>
>>>> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
>>>> cd bin/ and then run:
>>>>
>>>> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>>>
>>>> You can then get an idea of the command line parameters it takes:
>>>>
>>>> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>>> -a,--all Index all items in catalog
>>>> -cq,--catalogQuery <query> Not yet implemented
>>>> -d,--delete Delete items before indexing
>>>> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
>>>> -h,--help Print this message
>>>> -mf,--metFile <file> Index this met file
>>>> -o,--optimize Optimize the Solr index when done
>>>> -p,--product <productId> Product id to index
>>>> -su,--solrUrl <Solr URL> URL to the Solr server
>>>>
>>>> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>>>>
>>>> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>>>>
>>>> Cheers,
>>>> Tom
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Here's a basic summary of what I've discovered so far getting my metadata
into a Solr index...
SolrIndexer queries metadata for a single or multiple products from a File
Manager and put them into a Solr index. It does this by an http POST to
solr/update. Very cool.
Because it's xml on the wire, its really easy to tcpdump and view the
actual POST and see what the content is :). This is how I discovered that
the indexAll was not updating solr (see below).
Okay so for a simple test, I set up the following mapping, since these
fields already exist in the Solr schema.xml file:
map.ExperimentID=id
map.Observer=author
map.Description=description
Note: It seems a good idea to have an "id" field. If you look in the solr
scheme.xml file, I saw the following configuration by default:
<!-- Field to use to determine and enforce document uniqueness.
Unless this field is marked with required="false", it will be a
required field
-->
<uniqueKey>id</uniqueKey>
Okay, so I finally ran the SolrIndexer with the -all.
This piece of code (starting at line 209 of SolrIndexer.java) was never
executed. It's checking for metadata "UUID". I don't have this metadata
key.
if (metadata.getMetadata("UUID") != null) {
if (metadata.getMetadata("Deleted") == null
|| !"true".equals(metadata.getMetadata("Deleted"))) {
try {
server.add(this.getSolrDocument(metadata));
server.commit();
LOG.info("Indexed " + metadata.getMetadata("UUID"));
} catch (Exception e) {
LOG.severe("Could not index " + metadata.getMetadata(
"UUID")
+ " " + e.getMessage());
}
} else {
LOG.info("Skipping Deleted: " + metadata.getMetadata("UUID"
));
}
}
Question: Deleted looks optional, so no problem there, but what should I be
doing with UUID? I've changed this to be CAS.ProductId in the code, just to
check nothing else was causing a problem. I thought it would be worth
asking why UUID is specified...
I've tried the single product index, which works just fine.
Cheers,
Tom
On 30 March 2012 20:38, Mattmann, Chris A (388J) <
chris.a.mattmann@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I
> think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block
> in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in
> Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up
> later, but
> see if that helps for now.
>
> Cheers,
> Chris
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Thanks for the feedback. Makes perfect sense. I'll check it out tomorrow and let you know.
So far solr is looking promising, it even looks like I can integrate queries into python quiet easily via json. Double woot!
Sent from my iPhone
On 30 Mar 2012, at 8:38 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> No worries...based on your email below, did you update the Solr schema? I think you need
> to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
> to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
> specify which fields to send to Solr.
>
> Does that make sense? I can provide more detailed help and will follow up later, but
> see if that helps for now.
>
> Cheers,
> Chris
>
> On Mar 30, 2012, at 7:54 AM, Thomas Bennett wrote:
>
>> Hey Chris,
>>
>> Sorry for the vagueness of this email... I'm trying to get solr working, but I seem to be doing something wrong...
>>
>> Here's my indexer.properties file:
>>
>> # the URL path to Solr
>> config.solr.url=http://192.168.1.51:8983/solr
>>
>> # the URL path to the File Manager
>> config.filemgr.url=http://localhost:9101
>>
>> config.ref.DataDownloadRef=dataref
>>
>> # a set of products to map from File Manager
>> # terminology into Solr index doc field
>> # terminology
>> map.MimeType=mimetype
>> map.ReceivedTime=receivedtime
>> map.FileSize=filesize
>> map.FileName=filename
>> map.Antennas=antennas
>> map.Observer=observer
>> map.Description=description
>> map.StartTime=starttime
>>
>> I then ran:
>>
>> ~$ java -DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties -Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/ org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl http://localhost:9101
>>
>> Which returned things like:
>> ....
>> INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
>> 30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer indexAll
>> ...
>>
>> However when I tried to search from solr admin interface I tried a wild card search to find anything:
>>
>> *:*
>>
>> Also tried:
>>
>> observer:thomas which returned:
>> undefined field observer
>>
>> Looks like I'm not getting the index across properly.
>>
>> Any ideas....?
>>
>> Tom
>>
>> On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
>> Hey Chris,
>>
>> Thanks for the info :-) much obliged.
>>
>> Tom.
>>
>> Sent from my iPhone
>>
>> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Tom,
>>>
>>> No problem, here is the config sample:
>>>
>>> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
>>>
>>> HTH!
>>>
>>> Cheers,
>>> Chris
>>>
>>>
>>> On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
>>>
>>>> Hey Chris / Paul,
>>>>
>>>> Thanks for the info.
>>>>
>>>> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
>>>> cd bin/ and then run:
>>>>
>>>> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>>>
>>>> You can then get an idea of the command line parameters it takes:
>>>>
>>>> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>>> -a,--all Index all items in catalog
>>>> -cq,--catalogQuery <query> Not yet implemented
>>>> -d,--delete Delete items before indexing
>>>> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
>>>> -h,--help Print this message
>>>> -mf,--metFile <file> Index this met file
>>>> -o,--optimize Optimize the Solr index when done
>>>> -p,--product <productId> Product id to index
>>>> -su,--solrUrl <Solr URL> URL to the Solr server
>>>>
>>>> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>>>>
>>>> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>>>>
>>>> Cheers,
>>>> Tom
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>>
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
On Mar 30, 2012, at 7:54 AM, Thomas Bennett wrote:
> Hey Chris,
>
> Sorry for the vagueness of this email... I'm trying to get solr working, but I seem to be doing something wrong...
>
> Here's my indexer.properties file:
>
> # the URL path to Solr
> config.solr.url=http://192.168.1.51:8983/solr
>
> # the URL path to the File Manager
> config.filemgr.url=http://localhost:9101
>
> config.ref.DataDownloadRef=dataref
>
> # a set of products to map from File Manager
> # terminology into Solr index doc field
> # terminology
> map.MimeType=mimetype
> map.ReceivedTime=receivedtime
> map.FileSize=filesize
> map.FileName=filename
> map.Antennas=antennas
> map.Observer=observer
> map.Description=description
> map.StartTime=starttime
>
> I then ran:
>
> ~$ java -DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties -Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/ org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl http://localhost:9101
>
> Which returned things like:
> ....
> INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
> 30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer indexAll
> ...
>
> However when I tried to search from solr admin interface I tried a wild card search to find anything:
>
> *:*
>
> Also tried:
>
> observer:thomas which returned:
> undefined field observer
>
> Looks like I'm not getting the index across properly.
>
> Any ideas....?
>
> Tom
>
> On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
> Hey Chris,
>
> Thanks for the info :-) much obliged.
>
> Tom.
>
> Sent from my iPhone
>
> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
> > Hi Tom,
> >
> > No problem, here is the config sample:
> >
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
> >
> > HTH!
> >
> > Cheers,
> > Chris
> >
> >
> > On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> >
> >> Hey Chris / Paul,
> >>
> >> Thanks for the info.
> >>
> >> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
> >> cd bin/ and then run:
> >>
> >> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >>
> >> You can then get an idea of the command line parameters it takes:
> >>
> >> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >> -a,--all Index all items in catalog
> >> -cq,--catalogQuery <query> Not yet implemented
> >> -d,--delete Delete items before indexing
> >> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> >> -h,--help Print this message
> >> -mf,--metFile <file> Index this met file
> >> -o,--optimize Optimize the Solr index when done
> >> -p,--product <productId> Product id to index
> >> -su,--solrUrl <Solr URL> URL to the Solr server
> >>
> >> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
> >>
> >> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
> >>
> >> Cheers,
> >> Tom
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
No worries...based on your email below, did you update the Solr schema? I think you need
to tell it to allow for all fields, by uncommenting the dynamicField block in the schema.xml file
to say "allow all fields" to come through, or modify the schema.xml in Solr's conf directory to
specify which fields to send to Solr.
Does that make sense? I can provide more detailed help and will follow up later, but
see if that helps for now.
Cheers,
Chris
On Mar 30, 2012, at 7:54 AM, Thomas Bennett wrote:
> Hey Chris,
>
> Sorry for the vagueness of this email... I'm trying to get solr working, but I seem to be doing something wrong...
>
> Here's my indexer.properties file:
>
> # the URL path to Solr
> config.solr.url=http://192.168.1.51:8983/solr
>
> # the URL path to the File Manager
> config.filemgr.url=http://localhost:9101
>
> config.ref.DataDownloadRef=dataref
>
> # a set of products to map from File Manager
> # terminology into Solr index doc field
> # terminology
> map.MimeType=mimetype
> map.ReceivedTime=receivedtime
> map.FileSize=filesize
> map.FileName=filename
> map.Antennas=antennas
> map.Observer=observer
> map.Description=description
> map.StartTime=starttime
>
> I then ran:
>
> ~$ java -DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties -Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/ org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl http://localhost:9101
>
> Which returned things like:
> ....
> INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
> 30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer indexAll
> ...
>
> However when I tried to search from solr admin interface I tried a wild card search to find anything:
>
> *:*
>
> Also tried:
>
> observer:thomas which returned:
> undefined field observer
>
> Looks like I'm not getting the index across properly.
>
> Any ideas....?
>
> Tom
>
> On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
> Hey Chris,
>
> Thanks for the info :-) much obliged.
>
> Tom.
>
> Sent from my iPhone
>
> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
> > Hi Tom,
> >
> > No problem, here is the config sample:
> >
> > http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
> >
> > HTH!
> >
> > Cheers,
> > Chris
> >
> >
> > On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> >
> >> Hey Chris / Paul,
> >>
> >> Thanks for the info.
> >>
> >> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
> >> cd bin/ and then run:
> >>
> >> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >>
> >> You can then get an idea of the command line parameters it takes:
> >>
> >> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >> -a,--all Index all items in catalog
> >> -cq,--catalogQuery <query> Not yet implemented
> >> -d,--delete Delete items before indexing
> >> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> >> -h,--help Print this message
> >> -mf,--metFile <file> Index this met file
> >> -o,--optimize Optimize the Solr index when done
> >> -p,--product <productId> Product id to index
> >> -su,--solrUrl <Solr URL> URL to the Solr server
> >>
> >> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
> >>
> >> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
> >>
> >> Cheers,
> >> Tom
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Sorry for the vagueness of this email... I'm trying to get solr working,
but I seem to be doing something wrong...
Here's my indexer.properties file:
# the URL path to Solr
config.solr.url=http://192.168.1.51:8983/solr
# the URL path to the File Manager
config.filemgr.url=http://localhost:9101
config.ref.DataDownloadRef=dataref
# a set of products to map from File Manager
# terminology into Solr index doc field
# terminology
map.MimeType=mimetype
map.ReceivedTime=receivedtime
map.FileSize=filesize
map.FileName=filename
map.Antennas=antennas
map.Observer=observer
map.Description=description
map.StartTime=starttime
I then ran:
~$ java
-DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties
-Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/
org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl
http://localhost:9101
Which returned things like:
....
INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer
indexAll
...
However when I tried to search from solr admin interface I tried a wild
card search to find anything:
*:*
Also tried:
observer:thomas which returned:
undefined field observer
Looks like I'm not getting the index across properly.
Any ideas....?
Tom
On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
> Hey Chris,
>
> Thanks for the info :-) much obliged.
>
> Tom.
>
> Sent from my iPhone
>
> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Tom,
> >
> > No problem, here is the config sample:
> >
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
> >
> > HTH!
> >
> > Cheers,
> > Chris
> >
> >
> > On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> >
> >> Hey Chris / Paul,
> >>
> >> Thanks for the info.
> >>
> >> To get started, grab a fresh build (or build your own) copy of
> 0.4-SNAPSHOT. Then,
> >> cd bin/ and then run:
> >>
> >> java -Djava.ext.dirs=../lib
> org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >>
> >> You can then get an idea of the command line parameters it takes:
> >>
> >> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >> -a,--all Index all items in catalog
> >> -cq,--catalogQuery <query> Not yet implemented
> >> -d,--delete Delete items before indexing
> >> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> >> -h,--help Print this message
> >> -mf,--metFile <file> Index this met file
> >> -o,--optimize Optimize the Solr index when done
> >> -p,--product <productId> Product id to index
> >> -su,--solrUrl <Solr URL> URL to the Solr server
> >>
> >> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I
> can find some info on what is needed in the config file?
> >>
> >> I'm poking around the Solr docs to see if I can figure it out, but I
> thought I would just post the questions at the same time :-)
> >>
> >> Cheers,
> >> Tom
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Sorry for the vagueness of this email... I'm trying to get solr working,
but I seem to be doing something wrong...
Here's my indexer.properties file:
# the URL path to Solr
config.solr.url=http://192.168.1.51:8983/solr
# the URL path to the File Manager
config.filemgr.url=http://localhost:9101
config.ref.DataDownloadRef=dataref
# a set of products to map from File Manager
# terminology into Solr index doc field
# terminology
map.MimeType=mimetype
map.ReceivedTime=receivedtime
map.FileSize=filesize
map.FileName=filename
map.Antennas=antennas
map.Observer=observer
map.Description=description
map.StartTime=starttime
I then ran:
~$ java
-DSOLR_INDEXER_CONFIG=/var/kat/katconfig/static/oodt/cas-filemgr/policy/indexer.properties
-Djava.ext.dirs=/usr/local/oodt/cas-filemgr/lib/
org.apache.oodt.cas.filemgr.tools.SolrIndexer --all --fmUrl
http://localhost:9101
Which returned things like:
....
INFO: Looking up metadata for ProductId 19bcb4b8-7999-11e1-b581-8b771498975d
30 Mar 2012 4:32:37 PM org.apache.oodt.cas.filemgr.tools.SolrIndexer
indexAll
...
However when I tried to search from solr admin interface I tried a wild
card search to find anything:
*:*
Also tried:
observer:thomas which returned:
undefined field observer
Looks like I'm not getting the index across properly.
Any ideas....?
Tom
On 28 March 2012 20:05, Thomas Bennett <lm...@gmail.com> wrote:
> Hey Chris,
>
> Thanks for the info :-) much obliged.
>
> Tom.
>
> Sent from my iPhone
>
> On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <
> chris.a.mattmann@jpl.nasa.gov> wrote:
>
> > Hi Tom,
> >
> > No problem, here is the config sample:
> >
> >
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
> >
> > HTH!
> >
> > Cheers,
> > Chris
> >
> >
> > On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> >
> >> Hey Chris / Paul,
> >>
> >> Thanks for the info.
> >>
> >> To get started, grab a fresh build (or build your own) copy of
> 0.4-SNAPSHOT. Then,
> >> cd bin/ and then run:
> >>
> >> java -Djava.ext.dirs=../lib
> org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >>
> >> You can then get an idea of the command line parameters it takes:
> >>
> >> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> >> -a,--all Index all items in catalog
> >> -cq,--catalogQuery <query> Not yet implemented
> >> -d,--delete Delete items before indexing
> >> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> >> -h,--help Print this message
> >> -mf,--metFile <file> Index this met file
> >> -o,--optimize Optimize the Solr index when done
> >> -p,--product <productId> Product id to index
> >> -su,--solrUrl <Solr URL> URL to the Solr server
> >>
> >> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I
> can find some info on what is needed in the config file?
> >>
> >> I'm poking around the Solr docs to see if I can figure it out, but I
> thought I would just post the questions at the same time :-)
> >>
> >> Cheers,
> >> Tom
> >
> >
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Chris Mattmann, Ph.D.
> > Senior Computer Scientist
> > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> > Office: 171-266B, Mailstop: 171-246
> > Email: chris.a.mattmann@nasa.gov
> > WWW: http://sunset.usc.edu/~mattmann/
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> > Adjunct Assistant Professor, Computer Science Department
> > University of Southern California, Los Angeles, CA 90089 USA
> > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Thanks for the info :-) much obliged.
Tom.
Sent from my iPhone
On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> Hi Tom,
>
> No problem, here is the config sample:
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
>
> HTH!
>
> Cheers,
> Chris
>
>
> On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
>
>> Hey Chris / Paul,
>>
>> Thanks for the info.
>>
>> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
>> cd bin/ and then run:
>>
>> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>
>> You can then get an idea of the command line parameters it takes:
>>
>> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
>> -a,--all Index all items in catalog
>> -cq,--catalogQuery <query> Not yet implemented
>> -d,--delete Delete items before indexing
>> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
>> -h,--help Print this message
>> -mf,--metFile <file> Index this met file
>> -o,--optimize Optimize the Solr index when done
>> -p,--product <productId> Product id to index
>> -su,--solrUrl <Solr URL> URL to the Solr server
>>
>> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>>
>> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>>
>> Cheers,
>> Tom
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris,
Thanks for the info :-) much obliged.
Tom.
Sent from my iPhone
On 28 Mar 2012, at 5:16 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
> Hi Tom,
>
> No problem, here is the config sample:
>
> http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
>
> HTH!
>
> Cheers,
> Chris
>
>
> On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
>
>> Hey Chris / Paul,
>>
>> Thanks for the info.
>>
>> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
>> cd bin/ and then run:
>>
>> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>>
>> You can then get an idea of the command line parameters it takes:
>>
>> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
>> -a,--all Index all items in catalog
>> -cq,--catalogQuery <query> Not yet implemented
>> -d,--delete Delete items before indexing
>> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
>> -h,--help Print this message
>> -mf,--metFile <file> Index this met file
>> -o,--optimize Optimize the Solr index when done
>> -p,--product <productId> Product id to index
>> -su,--solrUrl <Solr URL> URL to the Solr server
>>
>> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>>
>> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>>
>> Cheers,
>> Tom
>
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW: http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
No problem, here is the config sample:
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
HTH!
Cheers,
Chris
On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> Hey Chris / Paul,
>
> Thanks for the info.
>
> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
> cd bin/ and then run:
>
> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>
> You can then get an idea of the command line parameters it takes:
>
> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> -a,--all Index all items in catalog
> -cq,--catalogQuery <query> Not yet implemented
> -d,--delete Delete items before indexing
> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> -h,--help Print this message
> -mf,--metFile <file> Index this met file
> -o,--optimize Optimize the Solr index when done
> -p,--product <productId> Product id to index
> -su,--solrUrl <Solr URL> URL to the Solr server
>
> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>
> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>
> Cheers,
> Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
No problem, here is the config sample:
http://svn.apache.org/repos/asf/oodt/trunk/filemgr/src/main/resources/indexer.properties
HTH!
Cheers,
Chris
On Mar 27, 2012, at 2:41 AM, Thomas Bennett wrote:
> Hey Chris / Paul,
>
> Thanks for the info.
>
> To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
> cd bin/ and then run:
>
> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>
> You can then get an idea of the command line parameters it takes:
>
> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> -a,--all Index all items in catalog
> -cq,--catalogQuery <query> Not yet implemented
> -d,--delete Delete items before indexing
> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> -h,--help Print this message
> -mf,--metFile <file> Index this met file
> -o,--optimize Optimize the Solr index when done
> -p,--product <productId> Product id to index
> -su,--solrUrl <Solr URL> URL to the Solr server
>
> The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can find some info on what is needed in the config file?
>
> I'm poking around the Solr docs to see if I can figure it out, but I thought I would just post the questions at the same time :-)
>
> Cheers,
> Tom
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris / Paul,
Thanks for the info.
To get started, grab a fresh build (or build your own) copy of
> 0.4-SNAPSHOT. Then,
> cd bin/ and then run:
>
> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>
> You can then get an idea of the command line parameters it takes:
>
> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> -a,--all Index all items in catalog
> -cq,--catalogQuery <query> Not yet implemented
> -d,--delete Delete items before indexing
> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> -h,--help Print this message
> -mf,--metFile <file> Index this met file
> -o,--optimize Optimize the Solr index when done
> -p,--product <productId> Product id to index
> -su,--solrUrl <Solr URL> URL to the Solr server
>
The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can
find some info on what is needed in the config file?
I'm poking around the Solr docs to see if I can figure it out, but I
thought I would just post the questions at the same time :-)
Cheers,
Tom
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hey Chris / Paul,
Thanks for the info.
To get started, grab a fresh build (or build your own) copy of
> 0.4-SNAPSHOT. Then,
> cd bin/ and then run:
>
> java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
>
> You can then get an idea of the command line parameters it takes:
>
> usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
> -a,--all Index all items in catalog
> -cq,--catalogQuery <query> Not yet implemented
> -d,--delete Delete items before indexing
> -fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
> -h,--help Print this message
> -mf,--metFile <file> Index this met file
> -o,--optimize Optimize the Solr index when done
> -p,--product <productId> Product id to index
> -su,--solrUrl <Solr URL> URL to the Solr server
>
The command is looking for a SOLR_INDEXER_CONFIG. Is there anywhere I can
find some info on what is needed in the config file?
I'm poking around the Solr docs to see if I can figure it out, but I
thought I would just post the questions at the same time :-)
Cheers,
Tom
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Feb 23, 2012, at 5:13 AM, Thomas Bennett wrote:
> Hi Chris,
>
> I'm just picking up this thread.
>
> I'm ready get Solr up and running :-)
Awesome!
>
> In reality, it the FM sees a star, we should really make sure we translate the query on the FM end into a WildcardQuery if we're using the LuceneCatalog. We should probably file a bug on this and fix at some point.
>
> In the meanwhile, one option if you need more complex searches is to take a look at the SolrIndexer that I just checked in:
>
> https://issues.apache.org/jira/browse/OODT-326
>
> Paul Ramirez wrote this tool and you can basically use it to dump the FM catalog into Solr directly and then query
> using Solr's syntax which is a bit more powerful than the FM's. The FM's Query Syntax is a trimmed down version
> suitable usually for production rules, for dumping metadata, and for staging files.
>
> At this point, a primer on getting started would be most useful.
To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
cd bin/ and then run:
java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
You can then get an idea of the command line parameters it takes:
usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
-a,--all Index all items in catalog
-cq,--catalogQuery <query> Not yet implemented
-d,--delete Delete items before indexing
-fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
-h,--help Print this message
-mf,--metFile <file> Index this met file
-o,--optimize Optimize the Solr index when done
-p,--product <productId> Product id to index
-su,--solrUrl <Solr URL> URL to the Solr server
Let me know if you need more info to start bootstrapping it.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Tom,
On Feb 23, 2012, at 5:13 AM, Thomas Bennett wrote:
> Hi Chris,
>
> I'm just picking up this thread.
>
> I'm ready get Solr up and running :-)
Awesome!
>
> In reality, it the FM sees a star, we should really make sure we translate the query on the FM end into a WildcardQuery if we're using the LuceneCatalog. We should probably file a bug on this and fix at some point.
>
> In the meanwhile, one option if you need more complex searches is to take a look at the SolrIndexer that I just checked in:
>
> https://issues.apache.org/jira/browse/OODT-326
>
> Paul Ramirez wrote this tool and you can basically use it to dump the FM catalog into Solr directly and then query
> using Solr's syntax which is a bit more powerful than the FM's. The FM's Query Syntax is a trimmed down version
> suitable usually for production rules, for dumping metadata, and for staging files.
>
> At this point, a primer on getting started would be most useful.
To get started, grab a fresh build (or build your own) copy of 0.4-SNAPSHOT. Then,
cd bin/ and then run:
java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.SolrIndexer
You can then get an idea of the command line parameters it takes:
usage: java org.apache.oodt.cas.filemgr.tools.SolrIndexer
-a,--all Index all items in catalog
-cq,--catalogQuery <query> Not yet implemented
-d,--delete Delete items before indexing
-fmu,--fmUrl <Filemgr URL> URL to the CAS FileManager
-h,--help Print this message
-mf,--metFile <file> Index this met file
-o,--optimize Optimize the Solr index when done
-p,--product <productId> Product id to index
-su,--solrUrl <Solr URL> URL to the Solr server
Let me know if you need more info to start bootstrapping it.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hi Chris,
I'm just picking up this thread.
I'm ready get Solr up and running :-)
In reality, it the FM sees a star, we should really make sure we translate
> the query on the FM end into a WildcardQuery if we're using the
> LuceneCatalog. We should probably file a bug on this and fix at some point.
>
> In the meanwhile, one option if you need more complex searches is to take
> a look at the SolrIndexer that I just checked in:
>
> https://issues.apache.org/jira/browse/OODT-326
>
> Paul Ramirez wrote this tool and you can basically use it to dump the FM
> catalog into Solr directly and then query
> using Solr's syntax which is a bit more powerful than the FM's. The FM's
> Query Syntax is a trimmed down version
> suitable usually for production rules, for dumping metadata, and for
> staging files.
>
At this point, a primer on getting started would be most useful.
Cheers,
Tom
Re: query syntax
Posted by Thomas Bennett <tb...@ska.ac.za>.
Hi,
Chris - thanks again for your reply and our Skype conversation.
To complete this email thread - it would seam that the query syntax allowed
by the File Manager is restricted and that the suggested approach is to use
the File Manager Query for doing searches from system level services (and
other components in OODT).
For user queries it would be better to have a nightly dump of the catalogue
and use Solr.
Cheers,
Tom
--
Thomas Bennett
SKA South Africa
Office : +2721 506 7341
Mobile : +2779 523 7105
Email : tbennett@ska.ac.za
Re: query syntax
Posted by Thomas Bennett <lm...@gmail.com>.
Hi Chris,
I'm just picking up this thread.
I'm ready get Solr up and running :-)
In reality, it the FM sees a star, we should really make sure we translate
> the query on the FM end into a WildcardQuery if we're using the
> LuceneCatalog. We should probably file a bug on this and fix at some point.
>
> In the meanwhile, one option if you need more complex searches is to take
> a look at the SolrIndexer that I just checked in:
>
> https://issues.apache.org/jira/browse/OODT-326
>
> Paul Ramirez wrote this tool and you can basically use it to dump the FM
> catalog into Solr directly and then query
> using Solr's syntax which is a bit more powerful than the FM's. The FM's
> Query Syntax is a trimmed down version
> suitable usually for production rules, for dumping metadata, and for
> staging files.
>
At this point, a primer on getting started would be most useful.
Cheers,
Tom
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
On Oct 19, 2011, at 6:22 AM, Thomas Bennett wrote:
> Hi,
>
> I'm trying out wildcards when using query_tool to run some queries on a lucene catalog and its throwing exceptions left right and center.
Brave man indeed! :-)
>
> Any help/pointers welcome.
>
> This query:
> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT Filename FROM KatFile WHERE Observer='ja*per'"
>
> Returns:
> Oct 19, 2011 3:19:53 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog paginateQuery
> WARNING: Query: [q=Observer:ja*per] for Product Type: [urn:kat:KatFile] returned no results
> java.lang.NullPointerException
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFileManager.java:602)
> [..snip...]
I think what you're seeing here is that we never really evolved the FreeTextQuery in the File Manager field-specific clauses
to be able to deal with '*'s. So, even though it is parsed correctly, the File Manager is translating the query into a TermQuery
in Lucene terminology, but with a '*' in it, which is a Lucene error. In reality, it the FM sees a star, we should really make sure
we translate the query on the FM end into a WildcardQuery if we're using the LuceneCatalog. We should probably file
a bug on this and fix at some point.
In the meanwhile, one option if you need more complex searches is to take a look at the SolrIndexer that I just checked in:
https://issues.apache.org/jira/browse/OODT-326
Paul Ramirez wrote this tool and you can basically use it to dump the FM catalog into Solr directly and then query
using Solr's syntax which is a bit more powerful than the FM's. The FM's Query Syntax is a trimmed down version
suitable usually for production rules, for dumping metadata, and for staging files.
If dumping to Solr is a bit much at this point, I can take a look at the query issue you're seeing (once you file it) and
give a hand towards trying to interpret the WildcardQuery clauses more correctly.
Thanks!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: query syntax
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Tom,
On Oct 19, 2011, at 6:22 AM, Thomas Bennett wrote:
> Hi,
>
> I'm trying out wildcards when using query_tool to run some queries on a lucene catalog and its throwing exceptions left right and center.
Brave man indeed! :-)
>
> Any help/pointers welcome.
>
> This query:
> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT Filename FROM KatFile WHERE Observer='ja*per'"
>
> Returns:
> Oct 19, 2011 3:19:53 PM org.apache.oodt.cas.filemgr.catalog.LuceneCatalog paginateQuery
> WARNING: Query: [q=Observer:ja*per] for Product Type: [urn:kat:KatFile] returned no results
> java.lang.NullPointerException
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFileManager.java:602)
> [..snip...]
I think what you're seeing here is that we never really evolved the FreeTextQuery in the File Manager field-specific clauses
to be able to deal with '*'s. So, even though it is parsed correctly, the File Manager is translating the query into a TermQuery
in Lucene terminology, but with a '*' in it, which is a Lucene error. In reality, it the FM sees a star, we should really make sure
we translate the query on the FM end into a WildcardQuery if we're using the LuceneCatalog. We should probably file
a bug on this and fix at some point.
In the meanwhile, one option if you need more complex searches is to take a look at the SolrIndexer that I just checked in:
https://issues.apache.org/jira/browse/OODT-326
Paul Ramirez wrote this tool and you can basically use it to dump the FM catalog into Solr directly and then query
using Solr's syntax which is a bit more powerful than the FM's. The FM's Query Syntax is a trimmed down version
suitable usually for production rules, for dumping metadata, and for staging files.
If dumping to Solr is a bit much at this point, I can take a look at the query issue you're seeing (once you file it) and
give a hand towards trying to interpret the WildcardQuery clauses more correctly.
Thanks!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++