You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oodt.apache.org by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> on 2012/05/10 22:28:22 UTC
Query Tool Bugs?
Hi All,
I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
Below is a command-line run of my query:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
I tried surrounding the entire condition with quotes, but still no luck:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
Ambiguous output redirect.
I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
at java.lang.StringBuffer.substring(StringBuffer.java:416)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Thanks,
Mike
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Thanks, reading the below:
On May 14, 2012, at 6:32 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> be why the tool might be having performance issues?
>>
>> Gotcha, that might help, yes. I was thinking: what is your repository
>> manager,
>> and catalog combination? If you are using e.g., a DataSourceCatalog,
>> with the XMLRepositoryManager, you'll need to turn on the quoteFields
>> option in the filemgr.properties for the DataSourceCatalog. This is
>> because,
>> in these scenarios, the identifier for elementIds is a string, compared
>> to a
>> number (which would be the case if you used the
>> DataSourceRepositoryManager --
>> the short answer there is don't, it's not as well maintained as the XML
>> one).
>
> My repository manager and catalog combination is defined as follows in the
> filemgr.properties:
>
> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
> toryManagerFactory
>
> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
> ogFactory
>
> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
Those look good. Will wait to see what Brian scopes...
Thanks!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
Thanks, reading the below:
On May 14, 2012, at 6:32 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> be why the tool might be having performance issues?
>>
>> Gotcha, that might help, yes. I was thinking: what is your repository
>> manager,
>> and catalog combination? If you are using e.g., a DataSourceCatalog,
>> with the XMLRepositoryManager, you'll need to turn on the quoteFields
>> option in the filemgr.properties for the DataSourceCatalog. This is
>> because,
>> in these scenarios, the identifier for elementIds is a string, compared
>> to a
>> number (which would be the case if you used the
>> DataSourceRepositoryManager --
>> the short answer there is don't, it's not as well maintained as the XML
>> one).
>
> My repository manager and catalog combination is defined as follows in the
> filemgr.properties:
>
> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
> toryManagerFactory
>
> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
> ogFactory
>
> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
Those look good. Will wait to see what Brian scopes...
Thanks!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>.
Hi Michael,
We're using Oracle.
-- cecilia
From: <Cayanan>, "Michael D (388J)" <mi...@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tuesday, May 22, 2012 7:50 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Cecilia,
The query works fine when you use either RangeBeginningDate or RangeBeginningTime. It's when you do an "AND" is the issue.
For your file manager, are you using mysql as the database? Or something else?
-Mike
From: "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 18:11:14 +0000
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>.
Hi Michael,
We're using Oracle.
-- cecilia
From: <Cayanan>, "Michael D (388J)" <mi...@jpl.nasa.gov>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Tuesday, May 22, 2012 7:50 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Cecilia,
The query works fine when you use either RangeBeginningDate or RangeBeginningTime. It's when you do an "AND" is the issue.
For your file manager, are you using mysql as the database? Or something else?
-Mike
From: "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 18:11:14 +0000
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Cecilia,
The query works fine when you use either RangeBeginningDate or RangeBeginningTime. It's when you do an "AND" is the issue.
For your file manager, are you using mysql as the database? Or something else?
-Mike
From: "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 18:11:14 +0000
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Cecilia,
The query works fine when you use either RangeBeginningDate or RangeBeginningTime. It's when you do an "AND" is the issue.
For your file manager, are you using mysql as the database? Or something else?
-Mike
From: "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 18:11:14 +0000
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>.
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
also... if putting the spaces still fails... i'm ready to bet this is a shell issue... instead of doing: ./query_tool --url .... put the arguments inside query_tool script so you can just run it like: ./query_tool
-brian
On May 17, 2012, at 05:51 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
mike can you post your SQL table CREATE statements
-brian
On May 21, 2012, at 12:12 PM, Brian Foster <ho...@mac.com> wrote:
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Chris/Brian,
Here are my CREATE SQL statements to create the tables in the mysql db:
I'm using a shell script to create tables for the various product types
that the File Manager will accept:
export product_types=\
"SPICE_TRAJ SPICE_ATT SPICE_ANT SPICE_EARTH SPICE_SCLK SPICE_FRAMES
SPICE_LEAP
SPICE_PCK SPICE_EFF SPICE_PTK \
InputConfig OutputConfig MetConfig Parameter \
SCAN NCDC_GSOD_WMO \
RunConfig Log QA Monitor\
RadarProductType RadiometerProductType L2_SM_ProductType L3_SM_ProductType
\
L0a_Radar L0a_Radiometer L0b_Radar
L1A_Radar L1B_S0_LoRes L1C_S0_HiRes L0b_Radiometer \
L1A_Radiometer L1B_TB L1C_TB L2_SM_A L2_SM_P L2_SM_AP \
L3_SM_A L3_SM_P L3_SM_AP L3_FT_A \
GenericFile SPDM_Type TimeBasedProductType GenericProductType \
DownlinkBasedProductType SMAP_CoreProductType HalfOrbitBasedProductType
DailyBasedProductType"
for i in ${product_types}
do
echo "CREATE TABLE ${i}_metadata"
echo "("
echo " product_id int NOT NULL,"
echo " element_id varchar(1000) NOT NULL,"
echo " metadata_value varchar(2500) NOT NULL"
echo ");"
echo "CREATE TABLE ${i}_reference"
echo "("
echo " product_id int NOT NULL,"
echo " product_orig_reference varchar(2000) NOT NULL,"
echo " product_datastore_reference varchar(2000),"
echo " product_reference_filesize int NOT NULL,"
echo " product_reference_mimetype varchar(50)"
echo ");"
done
CREATE TABLE `products` (
`product_id` int(11) NOT NULL auto_increment,
`product_structure` varchar(20) NOT NULL default '',
`product_type_id` varchar(255) NOT NULL default '',
`product_name` varchar(255) NOT NULL default '',
`product_transfer_status` varchar(255) NOT NULL default 'TRANSFERING',
PRIMARY KEY (`product_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Here is what the log is showing when running the following query tool
command:
%> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime >
'12:00:00.000'"
FINE: catalog query: executing: (SELECT DISTINCT product_id FROM
L0a_Radar_metadata WHERE element_id =
'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000')
INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE
element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value >
'2007-01-01') ORDER BY product_id DESC
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an
error in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'INTERSECT (SELECT
DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
ssorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
ctorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.Util.getInstance(Util.java:381)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3558)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3490)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2109)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2637)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2566)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1464)
at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatemen
t.java:205)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.paginateQuery(DataSou
rceCatalog.java:1965)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.query(DataSourceCatal
og.java:1179)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFil
eManager.java:600)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
9)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:680)
May 23, 2012 8:39:58 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
WARNING: Exception performing query. Message: You have an error in your
SQL syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id
FROM L0a_Radar_metadata WHERE element_id =' at line 1
May 23, 2012 8:39:58 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
SEVERE: Unable to rollback query transaction. Message: Can't call rollback
when autocommit=true
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: You have
an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'INTERSECT (SELECT
DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.paginateQuery(DataSou
rceCatalog.java:2020)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.query(DataSourceCatal
og.java:1179)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFil
eManager.java:600)
I did another quick test where I used an 'OR' instead of an 'AND' in my
query and I got a result back:
%> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
L0a_Radar WHERE RangeBeginningDate > '2007-01-01' OR RangeBeginningTime >
'12:00:00.000'"
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
FINE: catalog query: executing: (SELECT DISTINCT product_id FROM
L0a_Radar_metadata WHERE element_id =
'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000')
UNION (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id
= 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01')
ORDER BY product_id DESC
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog getProductById
FINE: getProductById: Executing: SELECT * FROM products WHERE product_id =
314
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog getMetadata
FINE: getMetadata: Executing: SELECT * FROM L0a_Radar_metadata WHERE
product_id = 314
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 1 results
I'm starting to convince myself that the issue here is that query tool
converts a user-given query with an 'AND' in a WHERE clause into a query
containing an INTERSECT syntax, which isn't supported in mysql. That's the
reason that an exception gets thrown.
I don't know if there is a way around this by configuring the file manager
differently (using something else other than XMLValidationLayer and/or
DataSourceCatalog).
As I've mentioned previously, SMAP will be moving to an oracle db very
soon, so this won't be an issue any longer as the INTERSECT syntax is
supported there. But this is an interesting finding and it's a lesson
learned. :)
-Mike
On 5/22/12 7:55 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>This is where quoteFields comes in.
>
>If you are using a combination of XMLValidationLayer with
>DataSourceCatalog, rather than
>the element IDs being numeric, they are strings.
>
>That's Brian is talking about I think. I you mentioned you were using
>quoteFields so
>it should take care of this, but seeing the SQL you're generating can
>help.
>
>Can you edit etc/logging.properties and turn the catalog level WAY up
>like to ALL?
>
>Then restart FM and tell us the queries that are printing from the
>logs/cas_filemgrX.log.Y.
>
>Cheers,
>Chris
>
>On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>
>> Hey Brian,
>>
>> element_id is set to a varchar. I'm using the attached shell script to
>>create the database tables. Since I'm using XMLValidationLayer, does
>>this mean that I should be creating a different set of tables? If so,
>>what would those be? I'm assuming the provided
>>cas-filemgr-schema-mysql.sql is used only if you're using
>>DataSourceValidationLayer...
>>
>> -Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 22:32:30 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike,
>>
>> i copied in your etc files and policy files into a newly deployed trunk
>>filemgr deployment... i then just commented out the code which talked to
>>sql driver so i could see the SQL which would be sent to the sql
>>driver... basically by doing that i've narrowed it down to something
>>with your SQL connection... did you change element_id to a varchar
>>instead of and int since you are using XMLValidationLayer... the provide
>>cas-filemgr-schema-mysql.sql is setup to work with
>>DataSourceValidationLayer... i think you are getting an error because
>>there are quotes around your element_id and the database is expecting a
>>integer (the quotes are correct in this case though, you DB is just
>>configured wrong)
>>
>> -brian
>>
>> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)"
>><mi...@jpl.nasa.gov> wrote:
>>
>>> Hey Brian,
>>>
>>> Hmmm...interesting.
>>>
>>> I believe I attached a filemgr.properties in an earlier e-mail. Did
>>>you use this in your configuration as well?
>>>
>>> Also, which version of the filemgr are you using on your computer? The
>>>trunk? Or the wengine-branch that you pointed me to earlier?
>>>
>>> I'm also using a mysql DB with the file manager. I don't think that
>>>should matter though in this case, but I could be wrong. I'm assuming
>>>you set up your file manager with some other db backend?
>>>
>>> Would it be possible to send me a tar package of your file manger
>>>deployment that you have on your computer? I want to see if I can just
>>>untar it on my computer, start it up and see if that works.
>>>
>>> Thanks again for all your help on this,
>>> Mike
>>>
>>> From: Brian Foster <ho...@mac.com>
>>> Reply-To: <us...@oodt.apache.org>
>>> Date: Mon, 21 May 2012 19:12:09 +0000
>>> To: <us...@oodt.apache.org>
>>> Subject: Re: Query Tool Bugs?
>>>
>>>
>>> hey mike... i'm kinda stumped... used your configuration on a filemgr
>>>on my computer and i get the following output:
>>>
>>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>>
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningTime' AND metadata_value >
>>>'12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningDate' AND metadata_value >
>>>'2007-01-01') ORDER BY product_id DESC
>>> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>>> INFO: Query returned 0 results
>>>
>>> There must be something special about your setup...
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>>
>>>>
>>>> Hey mike... i should have some time today to take a look into this
>>>>more than just throwing out guesses... i put a print statement in the
>>>>QueryTool code to see what the SqlParser was doing with the query...
>>>>
>>>> i ran the following:
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>>RangeBeginningTime > '12:00:00.000'"
>>>>
>>>> I got the following file manager query:
>>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>>RangeBeginningDate:[2007-01-01 TO null])
>>>>
>>>> so it appears that it doesn't have anything to do with shell parsing
>>>>(like i initially thought) or SqlParser... i will take a closer look
>>>>at your configuration today and let you know what i find.
>>>>
>>>> also something that has been know to cause problems in the past is
>>>>shell aliases... maybe take a look at those if you have any... in the
>>>>mean time i will try and get to the bottom of this.
>>>>
>>>> -brian
>>>>
>>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)"
>>>><mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> Yep, I tried the query with spaces before and after > and also
>>>>>surrounded each value with a single quote and it didn't work for me.
>>>>>I also put the query into the shell script as you've suggested and
>>>>>still no luck.
>>>>>
>>>>> The exception I'm getting is this:
>>>>>
>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpc
>>>>>ClientResponseProcessor.java:104)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcC
>>>>>lientResponseProcessor.java:71)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:7
>>>>>3)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:945)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.
>>>>>java:250)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>> Exception in thread "main"
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:951)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.
>>>>>java:250)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpc
>>>>>ClientResponseProcessor.java:104)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcC
>>>>>lientResponseProcessor.java:71)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:7
>>>>>3)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:945)
>>>>> ... 2 more
>>>>>
>>>>> Attached is the query tool script, which contains the query I'm
>>>>>trying. It appears that this is a bug. Since query tool has been
>>>>>deprecated in the trunk in favor of the super new file manager client
>>>>>tool, then I'll go ahead and file a JIRA issue for this against the
>>>>>file manager.
>>>>>
>>>>> In the meantime, I'll definitely look more into this and see if I
>>>>>can find out where the bug is at.
>>>>>
>>>>> Thanks for your help!
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> You putting a space between the operators? its hard to tell in
>>>>>emails, stuff gets normalized sometimes.
>>>>>
>>>>> RangeBeginningDate > '2007-01-01'
>>>>>
>>>>> Should be a space before and after >
>>>>>
>>>>> each value should start and end with single quote.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)"
>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> I checked out the wengine-branch filemgr and unfortunately, it
>>>>>>still didn't work. I tried a query using the wengine branch's
>>>>>>query_tool since this version of the filemgr-client didn't have the
>>>>>>ability to query.
>>>>>>
>>>>>> In any case, I took the liberty of diving into the Query Tool code
>>>>>>to see if I can find out more as to what is going on. So this is
>>>>>>what I found:
>>>>>>
>>>>>> Surrounding single quotes around each of the conditions got me
>>>>>>further into the code. So, when I entered the query as follows,
>>>>>>
>>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND
>>>>>>'RangeBeginningTime>12:00:00Z'
>>>>>>
>>>>>> The SQLParser class liked the syntax and it produced the following
>>>>>>post fix statement:
>>>>>>
>>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01',
>>>>>>'RangeBeginningTime>12:00:00Z', AND]
>>>>>>
>>>>>> I think from here, it creates some Boolean Query Criteria with the
>>>>>>following content:
>>>>>>
>>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO
>>>>>>null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>>
>>>>>> I noticed that there are unmatched single quotes around the date
>>>>>>and time values. Not sure how that came about. The tool still threw
>>>>>>an exception. I'm guessing it's because of the unmatched single
>>>>>>quotes.
>>>>>>
>>>>>> I tried surrounding the date and time values with single quotes as
>>>>>>well, but the SQLParser class didn't like that either:
>>>>>>
>>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND
>>>>>>'RangeBeginningTime>'12:00:00Z''
>>>>>>
>>>>>>
>>>>>> -Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> would you mind trying to run this query using the wengine-branch
>>>>>>filemgr
>>>>>>(https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemg
>>>>>>r/)... there are a lot of bug fixes floating around in both Lucene
>>>>>>and DataSource Catalogs in that branch... if it works with that then
>>>>>>we know it is a bug with trunk.
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)"
>>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Brian,
>>>>>>>
>>>>>>> Ya, I tried the query with and without the double quotes
>>>>>>>surrounding the where clause and they both didn't work for me.
>>>>>>>
>>>>>>> In any case, I've built the latest file manager in the trunk and
>>>>>>>it fixed that StringIndexOutofBoundsException issue I was seeing.
>>>>>>>Thanks!
>>>>>>>
>>>>>>> Out of curiosity, I tried using the latest File Manager Client
>>>>>>>tool with the query with multiple conditions to see if I could get
>>>>>>>a result, but it threw an exception as well:
>>>>>>>
>>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM
>>>>>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>>>>>RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>>> May 16, 2012 7:20:53 AM
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>>> INFO: Loading File Manager Configuration Properties from:
>>>>>>>[../etc/filemgr.properties]
>>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>>>version for the right syntax to use near 'INTERSECT (SELECT
>>>>>>>DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at
>>>>>>>line 1
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlR
>>>>>>>pcClientResponseProcessor.java:104)
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRp
>>>>>>>cClientResponseProcessor.java:71)
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java
>>>>>>>:73)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQu
>>>>>>>ery(XmlRpcFileManagerClient.java:990)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execut
>>>>>>>e(AbstractQueryCliAction.java:75)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:3
>>>>>>>31)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlR
>>>>>>>pcFileManagerClient.java:1323)
>>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat
>>>>>>>'null', and delimiter '
>>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null',
>>>>>>>endDateTimeMetKey 'null', priorityMetKey 'null', null' :
>>>>>>>java.lang.Exception:
>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>>>version for the right syntax to use near 'INTERSECT (SELECT
>>>>>>>DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at
>>>>>>>line 1
>>>>>>>
>>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>>> To: <us...@oodt.apache.org>
>>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>>
>>>>>>> hey mike,
>>>>>>>
>>>>>>> first pass over the policy files look good... however, is the way
>>>>>>>you posted the query the way you are running it?
>>>>>>>
>>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>>>>>>FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>>>RangeBeginningTime>'12:00:00.000Z'""
>>>>>>>
>>>>>>> you have an extra set of double quotes around the where clause...
>>>>>>>try removing those
>>>>>>>
>>>>>>> -brian
>>>>>>>
>>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)"
>>>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>>> Hey Chris,
>>>>>>>>
>>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>>
>>>>>>>> >Hey Mike,
>>>>>>>> >
>>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>>> >
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>> >>>>Failed
>>>>>>>> >>>> to perform complex query : You have an error in your SQL
>>>>>>>>syntax; check
>>>>>>>> >>>> the manual that corresponds to your MySQL server version for
>>>>>>>>the right
>>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id
>>>>>>>>FROM
>>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.comp
>>>>>>>>>>>>lexQuery
>>>>>>>> >>>>(X
>>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(Que
>>>>>>>>>>>>ryTool.j
>>>>>>>> >>>>av
>>>>>>>> >>>> a:251)
>>>>>>>> >>>> at
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java
>>>>>>>>>>>>:241)
>>>>>>>> >>>
>>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time
>>>>>>>>format? Looks
>>>>>>>> >>> like a partial one, missing the timezone do you think that
>>>>>>>>might
>>>>>>>> >>> affect ir?
>>>>>>>> >>
>>>>>>>> >> I talked with Rishi regarding this and he recommended that the
>>>>>>>>date and
>>>>>>>> >> time be split when performing a query. Reason being is that
>>>>>>>>the query
>>>>>>>> >>tool
>>>>>>>> >> blows up when trying to compare datetime values. He mentioned
>>>>>>>>that he
>>>>>>>> >> tried querying against ISO 8601 date/time values before and it
>>>>>>>>didn't
>>>>>>>> >>work
>>>>>>>> >> for him and the way around it was to split it up. I think
>>>>>>>>behind the
>>>>>>>> >> scenes, the query tool is actually doing an ascii comparison,
>>>>>>>>which
>>>>>>>> >>might
>>>>>>>> >> be why the tool might be having performance issues?
>>>>>>>> >
>>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your
>>>>>>>>repository
>>>>>>>> >manager,
>>>>>>>> >and catalog combination? If you are using e.g., a
>>>>>>>>DataSourceCatalog,
>>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the
>>>>>>>>quoteFields
>>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This
>>>>>>>>is
>>>>>>>> >because,
>>>>>>>> >in these scenarios, the identifier for elementIds is a string,
>>>>>>>>compared
>>>>>>>> >to a
>>>>>>>> >number (which would be the case if you used the
>>>>>>>> >DataSourceRepositoryManager --
>>>>>>>> >the short answer there is don't, it's not as well maintained as
>>>>>>>>the XML
>>>>>>>> >one).
>>>>>>>>
>>>>>>>> My repository manager and catalog combination is defined as
>>>>>>>>follows in the
>>>>>>>> filemgr.properties:
>>>>>>>>
>>>>>>>>
>>>>>>>>filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.X
>>>>>>>>MLReposi
>>>>>>>> toryManagerFactory
>>>>>>>>
>>>>>>>>
>>>>>>>>filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSou
>>>>>>>>rceCatal
>>>>>>>> ogFactory
>>>>>>>>
>>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> >
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>>>
>>>>>>>> >>>> I tried surrounding the entire condition with quotes, but
>>>>>>>>still no
>>>>>>>> >>>>luck:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>>> >>>> Ambiguous output redirect.
>>>>>>>> >>>>
>>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know
>>>>>>>>what the
>>>>>>>> >>>> tool is expecting.
>>>>>>>> >>>
>>>>>>>> >>> Did you check the code in SVN?
>>>>>>>> >>
>>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have
>>>>>>>>the code
>>>>>>>> >> checked out onto my local machine. I can certainly build the
>>>>>>>>trunk and
>>>>>>>> >>see
>>>>>>>> >> if I get the same results.
>>>>>>>> >
>>>>>>>> >I think there is a fix for something similar to this in the
>>>>>>>>trunk (as
>>>>>>>> >bfoster mentioned),
>>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>>> >problem, per
>>>>>>>> >above. Scope it out and let me know.
>>>>>>>>
>>>>>>>> Based on the properties specified above, the quoteFields option
>>>>>>>>is turned
>>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>>> It'll be interesting to see if Brian was able to find an error in
>>>>>>>>the
>>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>>
>>>>>>>> >
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>>>
>>>>>>>> >>>> My second issue that I'm running into is in regards to
>>>>>>>>querying of
>>>>>>>> >>>> dates. I tried the following query below and got the
>>>>>>>>following output:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>>> >>>> log4j:WARN See
>>>>>>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>>> >>>> for more info.
>>>>>>>> >>>> Exception in thread "main"
>>>>>>>>java.lang.StringIndexOutOfBoundsException:
>>>>>>>> >>>> String index out of range: -1
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder
>>>>>>>>>>>>.java:88
>>>>>>>> >>>>1)
>>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(Que
>>>>>>>>>>>>ryTool.j
>>>>>>>> >>>>av
>>>>>>>> >>>> a:255)
>>>>>>>> >>>> at
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java
>>>>>>>>>>>>:241)
>>>>>>>> >>>>
>>>>>>>> >>>> For this particular product, I have 1 product in my catalog
>>>>>>>>where the
>>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if
>>>>>>>>that factors
>>>>>>>> >>>> into why an exception is being thrown here. When I use an
>>>>>>>>earlier date
>>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>>> >>>> log4j:WARN See
>>>>>>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>>> >>>> for more info.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_R
>>>>>>>>>>>>adar,V20
>>>>>>>> >>>>51
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:
>>>>>>>>>>>>27:59.38
>>>>>>>> >>>>5-
>>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>>> >>>
>>>>>>>> >>> Interesting! Did you scope the code to see if there's a
>>>>>>>>RangeQuery
>>>>>>>> >>>issue?
>>>>>>>> >>>
>>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>>> >>
>>>>>>>> >> I haven't dived into the code, but will certainly do this as
>>>>>>>>SMAP will
>>>>>>>> >> need these capabilities. I will file a bug if it turns out
>>>>>>>>that this is
>>>>>>>> >> indeed a bug.
>>>>>>>> >
>>>>>>>> >Great Mike, thanks.
>>>>>>>> >
>>>>>>>> >Cheers,
>>>>>>>> >Chris
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>>> >Senior Computer Scientist
>>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >
>>>>>>>>
>>>>> #!/bin/sh
>>>>>
>>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool --url
>>>>>http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE
>>>>>RangeBeginningDate > '2007-01-01' AND RangeBeginningTime >
>>>>>'12:00:00.000'"
>> <createProductTypeSQL.sh>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Thanks to Brian, Chris, Cecilia, Rishi, and Cameron for your feedback on
this issue.
Cheers,
Mike
On 5/23/12 10:13 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Thanks Brian good job tracking this down.
>
>Thanks to Mike for finding it out -- I will head over to OODT-460 and
>think about
>the issue there.
>
>Mike, all: the stuff I showed Mike below about logging.properties and
>quoteFields
>though is a good illustration of how to see what SQL is being printed by
>the FM
>catalog and a good way/gotcha FAQ regarding if you have an error related
>to numerics and strings if you are combining XMLValidationLayer and
>DataSourceCatalog
>(for everything but MySQL apparently ;) ).
>
>Cheers,
>Chris
>
>On May 23, 2012, at 10:10 AM, Brian Foster wrote:
>
>> hey chris,
>>
>> this is an SQL compatiblity thing... MySQL doesn't support INTERSECT...
>>i created an issue for this fix: (OODT-460) DataSourceCatalog doesn't
>>work with MySQL
>>
>> -brian
>>
>> On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)"
>><ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> This is where quoteFields comes in.
>>>
>>> If you are using a combination of XMLValidationLayer with
>>>DataSourceCatalog, rather than
>>> the element IDs being numeric, they are strings.
>>>
>>> That's Brian is talking about I think. I you mentioned you were using
>>>quoteFields so
>>> it should take care of this, but seeing the SQL you're generating can
>>>help.
>>>
>>> Can you edit etc/logging.properties and turn the catalog level WAY up
>>>like to ALL?
>>>
>>> Then restart FM and tell us the queries that are printing from the
>>>logs/cas_filemgrX.log.Y.
>>>
>>> Cheers,
>>> Chris
>>>
>>> On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> > Hey Brian,
>>> >
>>> > element_id is set to a varchar. I'm using the attached shell script
>>>to create the database tables. Since I'm using XMLValidationLayer, does
>>>this mean that I should be creating a different set of tables? If so,
>>>what would those be? I'm assuming the provided
>>>cas-filemgr-schema-mysql.sql is used only if you're using
>>>DataSourceValidationLayer...
>>> >
>>> > -Mike
>>> >
>>> > From: Brian Foster <ho...@mac.com>
>>> > Reply-To: <us...@oodt.apache.org>
>>> > Date: Mon, 21 May 2012 22:32:30 +0000
>>> > To: <us...@oodt.apache.org>
>>> > Subject: Re: Query Tool Bugs?
>>> >
>>> >
>>> > hey mike,
>>> >
>>> > i copied in your etc files and policy files into a newly deployed
>>>trunk filemgr deployment... i then just commented out the code which
>>>talked to sql driver so i could see the SQL which would be sent to the
>>>sql driver... basically by doing that i've narrowed it down to
>>>something with your SQL connection... did you change element_id to a
>>>varchar instead of and int since you are using XMLValidationLayer...
>>>the provide cas-filemgr-schema-mysql.sql is setup to work with
>>>DataSourceValidationLayer... i think you are getting an error because
>>>there are quotes around your element_id and the database is expecting a
>>>integer (the quotes are correct in this case though, you DB is just
>>>configured wrong)
>>> >
>>> > -brian
>>> >
>>> > On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hey Brian,
>>> >>
>>> >> Hmmm...interesting.
>>> >>
>>> >> I believe I attached a filemgr.properties in an earlier e-mail. Did
>>>you use this in your configuration as well?
>>> >>
>>> >> Also, which version of the filemgr are you using on your computer?
>>>The trunk? Or the wengine-branch that you pointed me to earlier?
>>> >>
>>> >> I'm also using a mysql DB with the file manager. I don't think that
>>>should matter though in this case, but I could be wrong. I'm assuming
>>>you set up your file manager with some other db backend?
>>> >>
>>> >> Would it be possible to send me a tar package of your file manger
>>>deployment that you have on your computer? I want to see if I can just
>>>untar it on my computer, start it up and see if that works.
>>> >>
>>> >> Thanks again for all your help on this,
>>> >> Mike
>>> >>
>>> >> From: Brian Foster <ho...@mac.com>
>>> >> Reply-To: <us...@oodt.apache.org>
>>> >> Date: Mon, 21 May 2012 19:12:09 +0000
>>> >> To: <us...@oodt.apache.org>
>>> >> Subject: Re: Query Tool Bugs?
>>> >>
>>> >>
>>> >> hey mike... i'm kinda stumped... used your configuration on a
>>>filemgr on my computer and i get the following output:
>>> >>
>>> >> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>> >>
>>> >> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> >> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>>> >> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningTime' AND metadata_value >
>>>'12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningDate' AND metadata_value >
>>>'2007-01-01') ORDER BY product_id DESC
>>> >> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>>> >> INFO: Query returned 0 results
>>> >>
>>> >> There must be something special about your setup...
>>> >>
>>> >> -brian
>>> >>
>>> >> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>
>>>wrote:
>>> >>
>>> >>>
>>> >>> Hey mike... i should have some time today to take a look into this
>>>more than just throwing out guesses... i put a print statement in the
>>>QueryTool code to see what the SqlParser was doing with the query...
>>> >>>
>>> >>> i ran the following:
>>> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>> >>>
>>> >>> I got the following file manager query:
>>> >>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> >>>
>>> >>> so it appears that it doesn't have anything to do with shell
>>>parsing (like i initially thought) or SqlParser... i will take a closer
>>>look at your configuration today and let you know what i find.
>>> >>>
>>> >>> also something that has been know to cause problems in the past is
>>>shell aliases... maybe take a look at those if you have any... in the
>>>mean time i will try and get to the bottom of this.
>>> >>>
>>> >>> -brian
>>> >>>
>>> >>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>
>>> >>>> Hey Brian,
>>> >>>>
>>> >>>> Yep, I tried the query with spaces before and after > and also
>>>surrounded each value with a single quote and it didn't work for me. I
>>>also put the query into the shell script as you've suggested and still
>>>no luck.
>>> >>>>
>>> >>>> The exception I'm getting is this:
>>> >>>>
>>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:945)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>va:250)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>> Exception in thread "main"
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:951)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>va:250)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>> Caused by: org.apache.xmlrpc.XmlRpcException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:945)
>>> >>>> ... 2 more
>>> >>>>
>>> >>>> Attached is the query tool script, which contains the query I'm
>>>trying. It appears that this is a bug. Since query tool has been
>>>deprecated in the trunk in favor of the super new file manager client
>>>tool, then I'll go ahead and file a JIRA issue for this against the
>>>file manager.
>>> >>>>
>>> >>>> In the meantime, I'll definitely look more into this and see if I
>>>can find out where the bug is at.
>>> >>>>
>>> >>>> Thanks for your help!
>>> >>>>
>>> >>>> -Mike
>>> >>>>
>>> >>>> From: Brian Foster <ho...@mac.com>
>>> >>>> Reply-To: <us...@oodt.apache.org>
>>> >>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>> >>>> To: <us...@oodt.apache.org>
>>> >>>> Subject: Re: Query Tool Bugs?
>>> >>>>
>>> >>>>
>>> >>>> hey mike,
>>> >>>>
>>> >>>> You putting a space between the operators? its hard to tell in
>>>emails, stuff gets normalized sometimes.
>>> >>>>
>>> >>>> RangeBeginningDate > '2007-01-01'
>>> >>>>
>>> >>>> Should be a space before and after >
>>> >>>>
>>> >>>> each value should start and end with single quote.
>>> >>>>
>>> >>>> -brian
>>> >>>>
>>> >>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>
>>> >>>>> Hey Brian,
>>> >>>>>
>>> >>>>> I checked out the wengine-branch filemgr and unfortunately, it
>>>still didn't work. I tried a query using the wengine branch's
>>>query_tool since this version of the filemgr-client didn't have the
>>>ability to query.
>>> >>>>>
>>> >>>>> In any case, I took the liberty of diving into the Query Tool
>>>code to see if I can find out more as to what is going on. So this is
>>>what I found:
>>> >>>>>
>>> >>>>> Surrounding single quotes around each of the conditions got me
>>>further into the code. So, when I entered the query as follows,
>>> >>>>>
>>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01'
>>>AND 'RangeBeginningTime>12:00:00Z'
>>> >>>>>
>>> >>>>> The SQLParser class liked the syntax and it produced the
>>>following post fix statement:
>>> >>>>>
>>> >>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01',
>>>'RangeBeginningTime>12:00:00Z', AND]
>>> >>>>>
>>> >>>>> I think from here, it creates some Boolean Query Criteria with
>>>the following content:
>>> >>>>>
>>> >>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO
>>>null], 'RangeBeginningDate:[2007-01-01' TO null])
>>> >>>>>
>>> >>>>> I noticed that there are unmatched single quotes around the date
>>>and time values. Not sure how that came about. The tool still threw an
>>>exception. I'm guessing it's because of the unmatched single quotes.
>>> >>>>>
>>> >>>>> I tried surrounding the date and time values with single quotes
>>>as well, but the SQLParser class didn't like that either:
>>> >>>>>
>>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01''
>>>AND 'RangeBeginningTime>'12:00:00Z''
>>> >>>>>
>>> >>>>>
>>> >>>>> -Mike
>>> >>>>>
>>> >>>>> From: Brian Foster <ho...@mac.com>
>>> >>>>> Reply-To: <us...@oodt.apache.org>
>>> >>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>> >>>>> To: <us...@oodt.apache.org>
>>> >>>>> Subject: Re: Query Tool Bugs?
>>> >>>>>
>>> >>>>>
>>> >>>>> hey mike,
>>> >>>>>
>>> >>>>> would you mind trying to run this query using the wengine-branch
>>>filemgr
>>>(https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)
>>>... there are a lot of bug fixes floating around in both Lucene and
>>>DataSource Catalogs in that branch... if it works with that then we
>>>know it is a bug with trunk.
>>> >>>>>
>>> >>>>> -brian
>>> >>>>>
>>> >>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>>
>>> >>>>>> Hey Brian,
>>> >>>>>>
>>> >>>>>> Ya, I tried the query with and without the double quotes
>>>surrounding the where clause and they both didn't work for me.
>>> >>>>>>
>>> >>>>>> In any case, I've built the latest file manager in the trunk
>>>and it fixed that StringIndexOutofBoundsException issue I was seeing.
>>>Thanks!
>>> >>>>>>
>>> >>>>>> Out of curiosity, I tried using the latest File Manager Client
>>>tool with the query with multiple conditions to see if I could get a
>>>result, but it threw an exception as well:
>>> >>>>>>
>>> >>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>> >>>>>> May 16, 2012 7:20:53 AM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>> >>>>>> INFO: Loading File Manager Configuration Properties from:
>>>[../etc/filemgr.properties]
>>> >>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:990)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(Ab
>>>stractQueryCliAction.java:75)
>>> >>>>>> at
>>>org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>> >>>>>> at
>>>org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFi
>>>leManagerClient.java:1323)
>>> >>>>>> ERROR: Failed to perform sql query : sortBy 'null',
>>>outputFormat 'null', and delimiter '
>>> >>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null',
>>>endDateTimeMetKey 'null', priorityMetKey 'null', null' :
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>>
>>> >>>>>> Let me know if you'd like for me to file a bug for this.
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Mike
>>> >>>>>>
>>> >>>>>> From: Brian Foster <ho...@mac.com>
>>> >>>>>> Reply-To: <us...@oodt.apache.org>
>>> >>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>> >>>>>> To: <us...@oodt.apache.org>
>>> >>>>>> Subject: Re: Query Tool Bugs?
>>> >>>>>>
>>> >>>>>> hey mike,
>>> >>>>>>
>>> >>>>>> first pass over the policy files look good... however, is the
>>>way you posted the query the way you are running it?
>>> >>>>>>
>>> >>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT
>>>* FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'""
>>> >>>>>>
>>> >>>>>> you have an extra set of double quotes around the where
>>>clause... try removing those
>>> >>>>>>
>>> >>>>>> -brian
>>> >>>>>>
>>> >>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>>>
>>> >>>>>>> Hey Chris,
>>> >>>>>>>
>>> >>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>> >>>>>>> <ch...@jpl.nasa.gov> wrote:
>>> >>>>>>>
>>> >>>>>>> >Hey Mike,
>>> >>>>>>> >
>>> >>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>> >>>>>>> >
>>> >>>>>>> >>>>
>>> >>>>>>> >>>>
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> >>>>>>> >>>>Failed
>>> >>>>>>> >>>> to perform complex query : You have an error in your SQL
>>>syntax; check
>>> >>>>>>> >>>> the manual that corresponds to your MySQL server version
>>>for the right
>>> >>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id
>>>FROM
>>> >>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQu
>>>>>>>ery
>>> >>>>>>> >>>>(X
>>> >>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryToo
>>>>>>>l.j
>>> >>>>>>> >>>>av
>>> >>>>>>> >>>> a:251)
>>> >>>>>>> >>>> at
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>>>>> >>>
>>> >>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time
>>>format? Looks
>>> >>>>>>> >>> like a partial one, missing the timezone do you think that
>>>might
>>> >>>>>>> >>> affect ir?
>>> >>>>>>> >>
>>> >>>>>>> >> I talked with Rishi regarding this and he recommended that
>>>the date and
>>> >>>>>>> >> time be split when performing a query. Reason being is that
>>>the query
>>> >>>>>>> >>tool
>>> >>>>>>> >> blows up when trying to compare datetime values. He
>>>mentioned that he
>>> >>>>>>> >> tried querying against ISO 8601 date/time values before and
>>>it didn't
>>> >>>>>>> >>work
>>> >>>>>>> >> for him and the way around it was to split it up. I think
>>>behind the
>>> >>>>>>> >> scenes, the query tool is actually doing an ascii
>>>comparison, which
>>> >>>>>>> >>might
>>> >>>>>>> >> be why the tool might be having performance issues?
>>> >>>>>>> >
>>> >>>>>>> >Gotcha, that might help, yes. I was thinking: what is your
>>>repository
>>> >>>>>>> >manager,
>>> >>>>>>> >and catalog combination? If you are using e.g., a
>>>DataSourceCatalog,
>>> >>>>>>> >with the XMLRepositoryManager, you'll need to turn on the
>>>quoteFields
>>> >>>>>>> >option in the filemgr.properties for the DataSourceCatalog.
>>>This is
>>> >>>>>>> >because,
>>> >>>>>>> >in these scenarios, the identifier for elementIds is a
>>>string, compared
>>> >>>>>>> >to a
>>> >>>>>>> >number (which would be the case if you used the
>>> >>>>>>> >DataSourceRepositoryManager --
>>> >>>>>>> >the short answer there is don't, it's not as well maintained
>>>as the XML
>>> >>>>>>> >one).
>>> >>>>>>>
>>> >>>>>>> My repository manager and catalog combination is defined as
>>>follows in the
>>> >>>>>>> filemgr.properties:
>>> >>>>>>>
>>> >>>>>>>
>>>filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLRep
>>>osi
>>> >>>>>>> toryManagerFactory
>>> >>>>>>>
>>> >>>>>>>
>>>filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCa
>>>tal
>>> >>>>>>> ogFactory
>>> >>>>>>>
>>> >>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >>>
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> I tried surrounding the entire condition with quotes, but
>>>still no
>>> >>>>>>> >>>>luck:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>> >>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>> >>>>>>> >>>> Ambiguous output redirect.
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> I'm assuming this is a syntax thing, although I don't
>>>know what the
>>> >>>>>>> >>>> tool is expecting.
>>> >>>>>>> >>>
>>> >>>>>>> >>> Did you check the code in SVN?
>>> >>>>>>> >>
>>> >>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I
>>>have the code
>>> >>>>>>> >> checked out onto my local machine. I can certainly build
>>>the trunk and
>>> >>>>>>> >>see
>>> >>>>>>> >> if I get the same results.
>>> >>>>>>> >
>>> >>>>>>> >I think there is a fix for something similar to this in the
>>>trunk (as
>>> >>>>>>> >bfoster mentioned),
>>> >>>>>>> >but thinking about this more, I bet you're having the
>>>quoteFields
>>> >>>>>>> >problem, per
>>> >>>>>>> >above. Scope it out and let me know.
>>> >>>>>>>
>>> >>>>>>> Based on the properties specified above, the quoteFields
>>>option is turned
>>> >>>>>>> on. So I think the properties are set correctly, no?
>>> >>>>>>> It'll be interesting to see if Brian was able to find an error
>>>in the
>>> >>>>>>> policy files that I sent in the previous e-mail.
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >>>
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> My second issue that I'm running into is in regards to
>>>querying of
>>> >>>>>>> >>>> dates. I tried the following query below and got the
>>>following output:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>> >>>>>>> >>>> log4j:WARN See
>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> >>>>>>> >>>> for more info.
>>> >>>>>>> >>>> Exception in thread "main"
>>>java.lang.StringIndexOutOfBoundsException:
>>> >>>>>>> >>>> String index out of range: -1
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java
>>>>>>>:88
>>> >>>>>>> >>>>1)
>>> >>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryToo
>>>>>>>l.j
>>> >>>>>>> >>>>av
>>> >>>>>>> >>>> a:255)
>>> >>>>>>> >>>> at
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> For this particular product, I have 1 product in my
>>>catalog where the
>>> >>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if
>>>that factors
>>> >>>>>>> >>>> into why an exception is being thrown here. When I use an
>>>earlier date
>>> >>>>>>> >>>> on my query, the tool returns a result as expected:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>> >>>>>>> >>>> log4j:WARN See
>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> >>>>>>> >>>> for more info.
>>> >>>>>>> >>>>
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,
>>>>>>>V20
>>> >>>>>>> >>>>51
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59
>>>>>>>.38
>>> >>>>>>> >>>>5-
>>> >>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>> >>>>>>> >>>
>>> >>>>>>> >>> Interesting! Did you scope the code to see if there's a
>>>RangeQuery
>>> >>>>>>> >>>issue?
>>> >>>>>>> >>>
>>> >>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>> >>>>>>> >>
>>> >>>>>>> >> I haven't dived into the code, but will certainly do this
>>>as SMAP will
>>> >>>>>>> >> need these capabilities. I will file a bug if it turns out
>>>that this is
>>> >>>>>>> >> indeed a bug.
>>> >>>>>>> >
>>> >>>>>>> >Great Mike, thanks.
>>> >>>>>>> >
>>> >>>>>>> >Cheers,
>>> >>>>>>> >Chris
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> Mike
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >Chris Mattmann, Ph.D.
>>> >>>>>>> >Senior Computer Scientist
>>> >>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >>>>>>> >Office: 171-266B, Mailstop: 171-246
>>> >>>>>>> >Email: chris.a.mattmann@nasa.gov
>>> >>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>> >>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>> #!/bin/sh
>>> >>>>
>>> >>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib
>>>org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000
>>>--sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate >
>>>'2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>> > <createProductTypeSQL.sh>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Thanks to Brian, Chris, Cecilia, Rishi, and Cameron for your feedback on
this issue.
Cheers,
Mike
On 5/23/12 10:13 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Thanks Brian good job tracking this down.
>
>Thanks to Mike for finding it out -- I will head over to OODT-460 and
>think about
>the issue there.
>
>Mike, all: the stuff I showed Mike below about logging.properties and
>quoteFields
>though is a good illustration of how to see what SQL is being printed by
>the FM
>catalog and a good way/gotcha FAQ regarding if you have an error related
>to numerics and strings if you are combining XMLValidationLayer and
>DataSourceCatalog
>(for everything but MySQL apparently ;) ).
>
>Cheers,
>Chris
>
>On May 23, 2012, at 10:10 AM, Brian Foster wrote:
>
>> hey chris,
>>
>> this is an SQL compatiblity thing... MySQL doesn't support INTERSECT...
>>i created an issue for this fix: (OODT-460) DataSourceCatalog doesn't
>>work with MySQL
>>
>> -brian
>>
>> On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)"
>><ch...@jpl.nasa.gov> wrote:
>>
>>> Hi Mike,
>>>
>>> This is where quoteFields comes in.
>>>
>>> If you are using a combination of XMLValidationLayer with
>>>DataSourceCatalog, rather than
>>> the element IDs being numeric, they are strings.
>>>
>>> That's Brian is talking about I think. I you mentioned you were using
>>>quoteFields so
>>> it should take care of this, but seeing the SQL you're generating can
>>>help.
>>>
>>> Can you edit etc/logging.properties and turn the catalog level WAY up
>>>like to ALL?
>>>
>>> Then restart FM and tell us the queries that are printing from the
>>>logs/cas_filemgrX.log.Y.
>>>
>>> Cheers,
>>> Chris
>>>
>>> On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> > Hey Brian,
>>> >
>>> > element_id is set to a varchar. I'm using the attached shell script
>>>to create the database tables. Since I'm using XMLValidationLayer, does
>>>this mean that I should be creating a different set of tables? If so,
>>>what would those be? I'm assuming the provided
>>>cas-filemgr-schema-mysql.sql is used only if you're using
>>>DataSourceValidationLayer...
>>> >
>>> > -Mike
>>> >
>>> > From: Brian Foster <ho...@mac.com>
>>> > Reply-To: <us...@oodt.apache.org>
>>> > Date: Mon, 21 May 2012 22:32:30 +0000
>>> > To: <us...@oodt.apache.org>
>>> > Subject: Re: Query Tool Bugs?
>>> >
>>> >
>>> > hey mike,
>>> >
>>> > i copied in your etc files and policy files into a newly deployed
>>>trunk filemgr deployment... i then just commented out the code which
>>>talked to sql driver so i could see the SQL which would be sent to the
>>>sql driver... basically by doing that i've narrowed it down to
>>>something with your SQL connection... did you change element_id to a
>>>varchar instead of and int since you are using XMLValidationLayer...
>>>the provide cas-filemgr-schema-mysql.sql is setup to work with
>>>DataSourceValidationLayer... i think you are getting an error because
>>>there are quotes around your element_id and the database is expecting a
>>>integer (the quotes are correct in this case though, you DB is just
>>>configured wrong)
>>> >
>>> > -brian
>>> >
>>> > On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >
>>> >> Hey Brian,
>>> >>
>>> >> Hmmm...interesting.
>>> >>
>>> >> I believe I attached a filemgr.properties in an earlier e-mail. Did
>>>you use this in your configuration as well?
>>> >>
>>> >> Also, which version of the filemgr are you using on your computer?
>>>The trunk? Or the wengine-branch that you pointed me to earlier?
>>> >>
>>> >> I'm also using a mysql DB with the file manager. I don't think that
>>>should matter though in this case, but I could be wrong. I'm assuming
>>>you set up your file manager with some other db backend?
>>> >>
>>> >> Would it be possible to send me a tar package of your file manger
>>>deployment that you have on your computer? I want to see if I can just
>>>untar it on my computer, start it up and see if that works.
>>> >>
>>> >> Thanks again for all your help on this,
>>> >> Mike
>>> >>
>>> >> From: Brian Foster <ho...@mac.com>
>>> >> Reply-To: <us...@oodt.apache.org>
>>> >> Date: Mon, 21 May 2012 19:12:09 +0000
>>> >> To: <us...@oodt.apache.org>
>>> >> Subject: Re: Query Tool Bugs?
>>> >>
>>> >>
>>> >> hey mike... i'm kinda stumped... used your configuration on a
>>>filemgr on my computer and i get the following output:
>>> >>
>>> >> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>> >>
>>> >> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> >> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>>> >> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningTime' AND metadata_value >
>>>'12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningDate' AND metadata_value >
>>>'2007-01-01') ORDER BY product_id DESC
>>> >> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>>> >> INFO: Query returned 0 results
>>> >>
>>> >> There must be something special about your setup...
>>> >>
>>> >> -brian
>>> >>
>>> >> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>
>>>wrote:
>>> >>
>>> >>>
>>> >>> Hey mike... i should have some time today to take a look into this
>>>more than just throwing out guesses... i put a print statement in the
>>>QueryTool code to see what the SqlParser was doing with the query...
>>> >>>
>>> >>> i ran the following:
>>> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>> >>>
>>> >>> I got the following file manager query:
>>> >>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> >>>
>>> >>> so it appears that it doesn't have anything to do with shell
>>>parsing (like i initially thought) or SqlParser... i will take a closer
>>>look at your configuration today and let you know what i find.
>>> >>>
>>> >>> also something that has been know to cause problems in the past is
>>>shell aliases... maybe take a look at those if you have any... in the
>>>mean time i will try and get to the bottom of this.
>>> >>>
>>> >>> -brian
>>> >>>
>>> >>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>
>>> >>>> Hey Brian,
>>> >>>>
>>> >>>> Yep, I tried the query with spaces before and after > and also
>>>surrounded each value with a single quote and it didn't work for me. I
>>>also put the query into the shell script as you've suggested and still
>>>no luck.
>>> >>>>
>>> >>>> The exception I'm getting is this:
>>> >>>>
>>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:945)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>va:250)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>> Exception in thread "main"
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:951)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>va:250)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>> Caused by: org.apache.xmlrpc.XmlRpcException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:945)
>>> >>>> ... 2 more
>>> >>>>
>>> >>>> Attached is the query tool script, which contains the query I'm
>>>trying. It appears that this is a bug. Since query tool has been
>>>deprecated in the trunk in favor of the super new file manager client
>>>tool, then I'll go ahead and file a JIRA issue for this against the
>>>file manager.
>>> >>>>
>>> >>>> In the meantime, I'll definitely look more into this and see if I
>>>can find out where the bug is at.
>>> >>>>
>>> >>>> Thanks for your help!
>>> >>>>
>>> >>>> -Mike
>>> >>>>
>>> >>>> From: Brian Foster <ho...@mac.com>
>>> >>>> Reply-To: <us...@oodt.apache.org>
>>> >>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>> >>>> To: <us...@oodt.apache.org>
>>> >>>> Subject: Re: Query Tool Bugs?
>>> >>>>
>>> >>>>
>>> >>>> hey mike,
>>> >>>>
>>> >>>> You putting a space between the operators? its hard to tell in
>>>emails, stuff gets normalized sometimes.
>>> >>>>
>>> >>>> RangeBeginningDate > '2007-01-01'
>>> >>>>
>>> >>>> Should be a space before and after >
>>> >>>>
>>> >>>> each value should start and end with single quote.
>>> >>>>
>>> >>>> -brian
>>> >>>>
>>> >>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>
>>> >>>>> Hey Brian,
>>> >>>>>
>>> >>>>> I checked out the wengine-branch filemgr and unfortunately, it
>>>still didn't work. I tried a query using the wengine branch's
>>>query_tool since this version of the filemgr-client didn't have the
>>>ability to query.
>>> >>>>>
>>> >>>>> In any case, I took the liberty of diving into the Query Tool
>>>code to see if I can find out more as to what is going on. So this is
>>>what I found:
>>> >>>>>
>>> >>>>> Surrounding single quotes around each of the conditions got me
>>>further into the code. So, when I entered the query as follows,
>>> >>>>>
>>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01'
>>>AND 'RangeBeginningTime>12:00:00Z'
>>> >>>>>
>>> >>>>> The SQLParser class liked the syntax and it produced the
>>>following post fix statement:
>>> >>>>>
>>> >>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01',
>>>'RangeBeginningTime>12:00:00Z', AND]
>>> >>>>>
>>> >>>>> I think from here, it creates some Boolean Query Criteria with
>>>the following content:
>>> >>>>>
>>> >>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO
>>>null], 'RangeBeginningDate:[2007-01-01' TO null])
>>> >>>>>
>>> >>>>> I noticed that there are unmatched single quotes around the date
>>>and time values. Not sure how that came about. The tool still threw an
>>>exception. I'm guessing it's because of the unmatched single quotes.
>>> >>>>>
>>> >>>>> I tried surrounding the date and time values with single quotes
>>>as well, but the SQLParser class didn't like that either:
>>> >>>>>
>>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01''
>>>AND 'RangeBeginningTime>'12:00:00Z''
>>> >>>>>
>>> >>>>>
>>> >>>>> -Mike
>>> >>>>>
>>> >>>>> From: Brian Foster <ho...@mac.com>
>>> >>>>> Reply-To: <us...@oodt.apache.org>
>>> >>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>> >>>>> To: <us...@oodt.apache.org>
>>> >>>>> Subject: Re: Query Tool Bugs?
>>> >>>>>
>>> >>>>>
>>> >>>>> hey mike,
>>> >>>>>
>>> >>>>> would you mind trying to run this query using the wengine-branch
>>>filemgr
>>>(https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)
>>>... there are a lot of bug fixes floating around in both Lucene and
>>>DataSource Catalogs in that branch... if it works with that then we
>>>know it is a bug with trunk.
>>> >>>>>
>>> >>>>> -brian
>>> >>>>>
>>> >>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>>
>>> >>>>>> Hey Brian,
>>> >>>>>>
>>> >>>>>> Ya, I tried the query with and without the double quotes
>>>surrounding the where clause and they both didn't work for me.
>>> >>>>>>
>>> >>>>>> In any case, I've built the latest file manager in the trunk
>>>and it fixed that StringIndexOutofBoundsException issue I was seeing.
>>>Thanks!
>>> >>>>>>
>>> >>>>>> Out of curiosity, I tried using the latest File Manager Client
>>>tool with the query with multiple conditions to see if I could get a
>>>result, but it threw an exception as well:
>>> >>>>>>
>>> >>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>> >>>>>> May 16, 2012 7:20:53 AM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>> >>>>>> INFO: Loading File Manager Configuration Properties from:
>>>[../etc/filemgr.properties]
>>> >>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>ientResponseProcessor.java:104)
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>entResponseProcessor.java:71)
>>> >>>>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>XmlRpcFileManagerClient.java:990)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(Ab
>>>stractQueryCliAction.java:75)
>>> >>>>>> at
>>>org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>> >>>>>> at
>>>org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>> >>>>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFi
>>>leManagerClient.java:1323)
>>> >>>>>> ERROR: Failed to perform sql query : sortBy 'null',
>>>outputFormat 'null', and delimiter '
>>> >>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null',
>>>endDateTimeMetKey 'null', priorityMetKey 'null', null' :
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>>
>>> >>>>>> Let me know if you'd like for me to file a bug for this.
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Mike
>>> >>>>>>
>>> >>>>>> From: Brian Foster <ho...@mac.com>
>>> >>>>>> Reply-To: <us...@oodt.apache.org>
>>> >>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>> >>>>>> To: <us...@oodt.apache.org>
>>> >>>>>> Subject: Re: Query Tool Bugs?
>>> >>>>>>
>>> >>>>>> hey mike,
>>> >>>>>>
>>> >>>>>> first pass over the policy files look good... however, is the
>>>way you posted the query the way you are running it?
>>> >>>>>>
>>> >>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT
>>>* FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'""
>>> >>>>>>
>>> >>>>>> you have an extra set of double quotes around the where
>>>clause... try removing those
>>> >>>>>>
>>> >>>>>> -brian
>>> >>>>>>
>>> >>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)"
>>><mi...@jpl.nasa.gov> wrote:
>>> >>>>>>
>>> >>>>>>> Hey Chris,
>>> >>>>>>>
>>> >>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>> >>>>>>> <ch...@jpl.nasa.gov> wrote:
>>> >>>>>>>
>>> >>>>>>> >Hey Mike,
>>> >>>>>>> >
>>> >>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>> >>>>>>> >
>>> >>>>>>> >>>>
>>> >>>>>>> >>>>
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>> >>>>>>> >>>>Failed
>>> >>>>>>> >>>> to perform complex query : You have an error in your SQL
>>>syntax; check
>>> >>>>>>> >>>> the manual that corresponds to your MySQL server version
>>>for the right
>>> >>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id
>>>FROM
>>> >>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQu
>>>>>>>ery
>>> >>>>>>> >>>>(X
>>> >>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryToo
>>>>>>>l.j
>>> >>>>>>> >>>>av
>>> >>>>>>> >>>> a:251)
>>> >>>>>>> >>>> at
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>>>>> >>>
>>> >>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time
>>>format? Looks
>>> >>>>>>> >>> like a partial one, missing the timezone do you think that
>>>might
>>> >>>>>>> >>> affect ir?
>>> >>>>>>> >>
>>> >>>>>>> >> I talked with Rishi regarding this and he recommended that
>>>the date and
>>> >>>>>>> >> time be split when performing a query. Reason being is that
>>>the query
>>> >>>>>>> >>tool
>>> >>>>>>> >> blows up when trying to compare datetime values. He
>>>mentioned that he
>>> >>>>>>> >> tried querying against ISO 8601 date/time values before and
>>>it didn't
>>> >>>>>>> >>work
>>> >>>>>>> >> for him and the way around it was to split it up. I think
>>>behind the
>>> >>>>>>> >> scenes, the query tool is actually doing an ascii
>>>comparison, which
>>> >>>>>>> >>might
>>> >>>>>>> >> be why the tool might be having performance issues?
>>> >>>>>>> >
>>> >>>>>>> >Gotcha, that might help, yes. I was thinking: what is your
>>>repository
>>> >>>>>>> >manager,
>>> >>>>>>> >and catalog combination? If you are using e.g., a
>>>DataSourceCatalog,
>>> >>>>>>> >with the XMLRepositoryManager, you'll need to turn on the
>>>quoteFields
>>> >>>>>>> >option in the filemgr.properties for the DataSourceCatalog.
>>>This is
>>> >>>>>>> >because,
>>> >>>>>>> >in these scenarios, the identifier for elementIds is a
>>>string, compared
>>> >>>>>>> >to a
>>> >>>>>>> >number (which would be the case if you used the
>>> >>>>>>> >DataSourceRepositoryManager --
>>> >>>>>>> >the short answer there is don't, it's not as well maintained
>>>as the XML
>>> >>>>>>> >one).
>>> >>>>>>>
>>> >>>>>>> My repository manager and catalog combination is defined as
>>>follows in the
>>> >>>>>>> filemgr.properties:
>>> >>>>>>>
>>> >>>>>>>
>>>filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLRep
>>>osi
>>> >>>>>>> toryManagerFactory
>>> >>>>>>>
>>> >>>>>>>
>>>filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCa
>>>tal
>>> >>>>>>> ogFactory
>>> >>>>>>>
>>> >>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >>>
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> I tried surrounding the entire condition with quotes, but
>>>still no
>>> >>>>>>> >>>>luck:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>> >>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>> >>>>>>> >>>> Ambiguous output redirect.
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> I'm assuming this is a syntax thing, although I don't
>>>know what the
>>> >>>>>>> >>>> tool is expecting.
>>> >>>>>>> >>>
>>> >>>>>>> >>> Did you check the code in SVN?
>>> >>>>>>> >>
>>> >>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I
>>>have the code
>>> >>>>>>> >> checked out onto my local machine. I can certainly build
>>>the trunk and
>>> >>>>>>> >>see
>>> >>>>>>> >> if I get the same results.
>>> >>>>>>> >
>>> >>>>>>> >I think there is a fix for something similar to this in the
>>>trunk (as
>>> >>>>>>> >bfoster mentioned),
>>> >>>>>>> >but thinking about this more, I bet you're having the
>>>quoteFields
>>> >>>>>>> >problem, per
>>> >>>>>>> >above. Scope it out and let me know.
>>> >>>>>>>
>>> >>>>>>> Based on the properties specified above, the quoteFields
>>>option is turned
>>> >>>>>>> on. So I think the properties are set correctly, no?
>>> >>>>>>> It'll be interesting to see if Brian was able to find an error
>>>in the
>>> >>>>>>> policy files that I sent in the previous e-mail.
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>> >>
>>> >>>>>>> >>>
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> My second issue that I'm running into is in regards to
>>>querying of
>>> >>>>>>> >>>> dates. I tried the following query below and got the
>>>following output:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>> >>>>>>> >>>> log4j:WARN See
>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> >>>>>>> >>>> for more info.
>>> >>>>>>> >>>> Exception in thread "main"
>>>java.lang.StringIndexOutOfBoundsException:
>>> >>>>>>> >>>> String index out of range: -1
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java
>>>>>>>:88
>>> >>>>>>> >>>>1)
>>> >>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> >>>>>>> >>>> at
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryToo
>>>>>>>l.j
>>> >>>>>>> >>>>av
>>> >>>>>>> >>>> a:255)
>>> >>>>>>> >>>> at
>>> >>>>>>>
>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> For this particular product, I have 1 product in my
>>>catalog where the
>>> >>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if
>>>that factors
>>> >>>>>>> >>>> into why an exception is being thrown here. When I use an
>>>earlier date
>>> >>>>>>> >>>> on my query, the tool returns a result as expected:
>>> >>>>>>> >>>>
>>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>"SELECT * FROM
>>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>> >>>>>>> >>>> log4j:WARN See
>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> >>>>>>> >>>> for more info.
>>> >>>>>>> >>>>
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,
>>>>>>>V20
>>> >>>>>>> >>>>51
>>> >>>>>>> >>>>
>>> >>>>>>>
>>>>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59
>>>>>>>.38
>>> >>>>>>> >>>>5-
>>> >>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>> >>>>>>> >>>
>>> >>>>>>> >>> Interesting! Did you scope the code to see if there's a
>>>RangeQuery
>>> >>>>>>> >>>issue?
>>> >>>>>>> >>>
>>> >>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>> >>>>>>> >>
>>> >>>>>>> >> I haven't dived into the code, but will certainly do this
>>>as SMAP will
>>> >>>>>>> >> need these capabilities. I will file a bug if it turns out
>>>that this is
>>> >>>>>>> >> indeed a bug.
>>> >>>>>>> >
>>> >>>>>>> >Great Mike, thanks.
>>> >>>>>>> >
>>> >>>>>>> >Cheers,
>>> >>>>>>> >Chris
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> Mike
>>> >>>>>>>
>>> >>>>>>> >
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >Chris Mattmann, Ph.D.
>>> >>>>>>> >Senior Computer Scientist
>>> >>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> >>>>>>> >Office: 171-266B, Mailstop: 171-246
>>> >>>>>>> >Email: chris.a.mattmann@nasa.gov
>>> >>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>> >>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>> >>>>>>>
>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> >>>>>>> >
>>> >>>>>>>
>>> >>>> #!/bin/sh
>>> >>>>
>>> >>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib
>>>org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000
>>>--sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate >
>>>'2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>> > <createProductTypeSQL.sh>
>>>
>>>
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Chris Mattmann, Ph.D.
>>> Senior Computer Scientist
>>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>> Office: 171-266B, Mailstop: 171-246
>>> Email: chris.a.mattmann@nasa.gov
>>> WWW: http://sunset.usc.edu/~mattmann/
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> Adjunct Assistant Professor, Computer Science Department
>>> University of Southern California, Los Angeles, CA 90089 USA
>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Brian good job tracking this down.
Thanks to Mike for finding it out -- I will head over to OODT-460 and think about
the issue there.
Mike, all: the stuff I showed Mike below about logging.properties and quoteFields
though is a good illustration of how to see what SQL is being printed by the FM
catalog and a good way/gotcha FAQ regarding if you have an error related
to numerics and strings if you are combining XMLValidationLayer and DataSourceCatalog
(for everything but MySQL apparently ;) ).
Cheers,
Chris
On May 23, 2012, at 10:10 AM, Brian Foster wrote:
> hey chris,
>
> this is an SQL compatiblity thing... MySQL doesn't support INTERSECT... i created an issue for this fix: (OODT-460) DataSourceCatalog doesn't work with MySQL
>
> -brian
>
> On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> This is where quoteFields comes in.
>>
>> If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
>> the element IDs being numeric, they are strings.
>>
>> That's Brian is talking about I think. I you mentioned you were using quoteFields so
>> it should take care of this, but seeing the SQL you're generating can help.
>>
>> Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
>>
>> Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
>>
>> Cheers,
>> Chris
>>
>> On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>>
>> > Hey Brian,
>> >
>> > element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>> >
>> > -Mike
>> >
>> > From: Brian Foster <ho...@mac.com>
>> > Reply-To: <us...@oodt.apache.org>
>> > Date: Mon, 21 May 2012 22:32:30 +0000
>> > To: <us...@oodt.apache.org>
>> > Subject: Re: Query Tool Bugs?
>> >
>> >
>> > hey mike,
>> >
>> > i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>> >
>> > -brian
>> >
>> > On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >
>> >> Hey Brian,
>> >>
>> >> Hmmm...interesting.
>> >>
>> >> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>> >>
>> >> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>> >>
>> >> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>> >>
>> >> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>> >>
>> >> Thanks again for all your help on this,
>> >> Mike
>> >>
>> >> From: Brian Foster <ho...@mac.com>
>> >> Reply-To: <us...@oodt.apache.org>
>> >> Date: Mon, 21 May 2012 19:12:09 +0000
>> >> To: <us...@oodt.apache.org>
>> >> Subject: Re: Query Tool Bugs?
>> >>
>> >>
>> >> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>> >>
>> >> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> >>
>> >> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> >> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> >> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> >> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> >> INFO: Query returned 0 results
>> >>
>> >> There must be something special about your setup...
>> >>
>> >> -brian
>> >>
>> >> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>> >>
>> >>>
>> >>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>> >>>
>> >>> i ran the following:
>> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> >>>
>> >>> I got the following file manager query:
>> >>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> >>>
>> >>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>> >>>
>> >>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>> >>>
>> >>> -brian
>> >>>
>> >>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>
>> >>>> Hey Brian,
>> >>>>
>> >>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>> >>>>
>> >>>> The exception I'm getting is this:
>> >>>>
>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>> >>>> ... 2 more
>> >>>>
>> >>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>> >>>>
>> >>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>> >>>>
>> >>>> Thanks for your help!
>> >>>>
>> >>>> -Mike
>> >>>>
>> >>>> From: Brian Foster <ho...@mac.com>
>> >>>> Reply-To: <us...@oodt.apache.org>
>> >>>> Date: Fri, 18 May 2012 00:51:07 +0000
>> >>>> To: <us...@oodt.apache.org>
>> >>>> Subject: Re: Query Tool Bugs?
>> >>>>
>> >>>>
>> >>>> hey mike,
>> >>>>
>> >>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>> >>>>
>> >>>> RangeBeginningDate > '2007-01-01'
>> >>>>
>> >>>> Should be a space before and after >
>> >>>>
>> >>>> each value should start and end with single quote.
>> >>>>
>> >>>> -brian
>> >>>>
>> >>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>
>> >>>>> Hey Brian,
>> >>>>>
>> >>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>> >>>>>
>> >>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>> >>>>>
>> >>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>> >>>>>
>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>> >>>>>
>> >>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>> >>>>>
>> >>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>> >>>>>
>> >>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>> >>>>>
>> >>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>> >>>>>
>> >>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>> >>>>>
>> >>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>> >>>>>
>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>> >>>>>
>> >>>>>
>> >>>>> -Mike
>> >>>>>
>> >>>>> From: Brian Foster <ho...@mac.com>
>> >>>>> Reply-To: <us...@oodt.apache.org>
>> >>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>> >>>>> To: <us...@oodt.apache.org>
>> >>>>> Subject: Re: Query Tool Bugs?
>> >>>>>
>> >>>>>
>> >>>>> hey mike,
>> >>>>>
>> >>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>> >>>>>
>> >>>>> -brian
>> >>>>>
>> >>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>>
>> >>>>>> Hey Brian,
>> >>>>>>
>> >>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>> >>>>>>
>> >>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>> >>>>>>
>> >>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>> >>>>>>
>> >>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>> >>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>> >>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>> >>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>> >>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>> >>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>> >>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>> >>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>>
>> >>>>>> Let me know if you'd like for me to file a bug for this.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Mike
>> >>>>>>
>> >>>>>> From: Brian Foster <ho...@mac.com>
>> >>>>>> Reply-To: <us...@oodt.apache.org>
>> >>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>> >>>>>> To: <us...@oodt.apache.org>
>> >>>>>> Subject: Re: Query Tool Bugs?
>> >>>>>>
>> >>>>>> hey mike,
>> >>>>>>
>> >>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>> >>>>>>
>> >>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>> >>>>>>
>> >>>>>> you have an extra set of double quotes around the where clause... try removing those
>> >>>>>>
>> >>>>>> -brian
>> >>>>>>
>> >>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>>>
>> >>>>>>> Hey Chris,
>> >>>>>>>
>> >>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>> >>>>>>> <ch...@jpl.nasa.gov> wrote:
>> >>>>>>>
>> >>>>>>> >Hey Mike,
>> >>>>>>> >
>> >>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>> >>>>>>> >
>> >>>>>>> >>>>
>> >>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> >>>>>>> >>>>Failed
>> >>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>> >>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>> >>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>> >>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>> >>>>>>> >>>>(X
>> >>>>>>> >>>> mlRpcFileManagerClient.java:958)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>> >>>>>>> >>>>av
>> >>>>>>> >>>> a:251)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>>>>> >>>
>> >>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>> >>>>>>> >>> like a partial one, missing the timezone do you think that might
>> >>>>>>> >>> affect ir?
>> >>>>>>> >>
>> >>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>> >>>>>>> >> time be split when performing a query. Reason being is that the query
>> >>>>>>> >>tool
>> >>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>> >>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>> >>>>>>> >>work
>> >>>>>>> >> for him and the way around it was to split it up. I think behind the
>> >>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>> >>>>>>> >>might
>> >>>>>>> >> be why the tool might be having performance issues?
>> >>>>>>> >
>> >>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>> >>>>>>> >manager,
>> >>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>> >>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>> >>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>> >>>>>>> >because,
>> >>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>> >>>>>>> >to a
>> >>>>>>> >number (which would be the case if you used the
>> >>>>>>> >DataSourceRepositoryManager --
>> >>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>> >>>>>>> >one).
>> >>>>>>>
>> >>>>>>> My repository manager and catalog combination is defined as follows in the
>> >>>>>>> filemgr.properties:
>> >>>>>>>
>> >>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>> >>>>>>> toryManagerFactory
>> >>>>>>>
>> >>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>> >>>>>>> ogFactory
>> >>>>>>>
>> >>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >>
>> >>>>>>> >>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>> >>>>>>> >>>>luck:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>> >>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>> >>>>>>> >>>> Ambiguous output redirect.
>> >>>>>>> >>>>
>> >>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>> >>>>>>> >>>> tool is expecting.
>> >>>>>>> >>>
>> >>>>>>> >>> Did you check the code in SVN?
>> >>>>>>> >>
>> >>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> >>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>> >>>>>>> >>see
>> >>>>>>> >> if I get the same results.
>> >>>>>>> >
>> >>>>>>> >I think there is a fix for something similar to this in the trunk (as
>> >>>>>>> >bfoster mentioned),
>> >>>>>>> >but thinking about this more, I bet you're having the quoteFields
>> >>>>>>> >problem, per
>> >>>>>>> >above. Scope it out and let me know.
>> >>>>>>>
>> >>>>>>> Based on the properties specified above, the quoteFields option is turned
>> >>>>>>> on. So I think the properties are set correctly, no?
>> >>>>>>> It'll be interesting to see if Brian was able to find an error in the
>> >>>>>>> policy files that I sent in the previous e-mail.
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >>
>> >>>>>>> >>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>> >>>>>>> >>>> dates. I tried the following query below and got the following output:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>> >>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>> >>>>>>> >>>> for more info.
>> >>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>> >>>>>>> >>>> String index out of range: -1
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>> >>>>>>> >>>>1)
>> >>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>> >>>>>>> >>>>av
>> >>>>>>> >>>> a:255)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>>>>> >>>>
>> >>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>> >>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>> >>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>> >>>>>>> >>>> on my query, the tool returns a result as expected:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>> >>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>> >>>>>>> >>>> for more info.
>> >>>>>>> >>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>> >>>>>>> >>>>51
>> >>>>>>> >>>>
>> >>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>> >>>>>>> >>>>5-
>> >>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>> >>>>>>> >>>
>> >>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>> >>>>>>> >>>issue?
>> >>>>>>> >>>
>> >>>>>>> >>> Feel free to file a bug and would love you to investigate!
>> >>>>>>> >>
>> >>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>> >>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>> >>>>>>> >> indeed a bug.
>> >>>>>>> >
>> >>>>>>> >Great Mike, thanks.
>> >>>>>>> >
>> >>>>>>> >Cheers,
>> >>>>>>> >Chris
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Mike
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >Chris Mattmann, Ph.D.
>> >>>>>>> >Senior Computer Scientist
>> >>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>>>>>> >Office: 171-266B, Mailstop: 171-246
>> >>>>>>> >Email: chris.a.mattmann@nasa.gov
>> >>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >Adjunct Assistant Professor, Computer Science Department
>> >>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >
>> >>>>>>>
>> >>>> #!/bin/sh
>> >>>>
>> >>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> > <createProductTypeSQL.sh>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Thanks Brian good job tracking this down.
Thanks to Mike for finding it out -- I will head over to OODT-460 and think about
the issue there.
Mike, all: the stuff I showed Mike below about logging.properties and quoteFields
though is a good illustration of how to see what SQL is being printed by the FM
catalog and a good way/gotcha FAQ regarding if you have an error related
to numerics and strings if you are combining XMLValidationLayer and DataSourceCatalog
(for everything but MySQL apparently ;) ).
Cheers,
Chris
On May 23, 2012, at 10:10 AM, Brian Foster wrote:
> hey chris,
>
> this is an SQL compatiblity thing... MySQL doesn't support INTERSECT... i created an issue for this fix: (OODT-460) DataSourceCatalog doesn't work with MySQL
>
> -brian
>
> On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
>
>> Hi Mike,
>>
>> This is where quoteFields comes in.
>>
>> If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
>> the element IDs being numeric, they are strings.
>>
>> That's Brian is talking about I think. I you mentioned you were using quoteFields so
>> it should take care of this, but seeing the SQL you're generating can help.
>>
>> Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
>>
>> Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
>>
>> Cheers,
>> Chris
>>
>> On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>>
>> > Hey Brian,
>> >
>> > element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>> >
>> > -Mike
>> >
>> > From: Brian Foster <ho...@mac.com>
>> > Reply-To: <us...@oodt.apache.org>
>> > Date: Mon, 21 May 2012 22:32:30 +0000
>> > To: <us...@oodt.apache.org>
>> > Subject: Re: Query Tool Bugs?
>> >
>> >
>> > hey mike,
>> >
>> > i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>> >
>> > -brian
>> >
>> > On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >
>> >> Hey Brian,
>> >>
>> >> Hmmm...interesting.
>> >>
>> >> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>> >>
>> >> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>> >>
>> >> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>> >>
>> >> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>> >>
>> >> Thanks again for all your help on this,
>> >> Mike
>> >>
>> >> From: Brian Foster <ho...@mac.com>
>> >> Reply-To: <us...@oodt.apache.org>
>> >> Date: Mon, 21 May 2012 19:12:09 +0000
>> >> To: <us...@oodt.apache.org>
>> >> Subject: Re: Query Tool Bugs?
>> >>
>> >>
>> >> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>> >>
>> >> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> >>
>> >> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> >> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> >> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> >> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> >> INFO: Query returned 0 results
>> >>
>> >> There must be something special about your setup...
>> >>
>> >> -brian
>> >>
>> >> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>> >>
>> >>>
>> >>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>> >>>
>> >>> i ran the following:
>> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> >>>
>> >>> I got the following file manager query:
>> >>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> >>>
>> >>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>> >>>
>> >>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>> >>>
>> >>> -brian
>> >>>
>> >>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>
>> >>>> Hey Brian,
>> >>>>
>> >>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>> >>>>
>> >>>> The exception I'm getting is this:
>> >>>>
>> >>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>> >>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>> >>>> ... 2 more
>> >>>>
>> >>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>> >>>>
>> >>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>> >>>>
>> >>>> Thanks for your help!
>> >>>>
>> >>>> -Mike
>> >>>>
>> >>>> From: Brian Foster <ho...@mac.com>
>> >>>> Reply-To: <us...@oodt.apache.org>
>> >>>> Date: Fri, 18 May 2012 00:51:07 +0000
>> >>>> To: <us...@oodt.apache.org>
>> >>>> Subject: Re: Query Tool Bugs?
>> >>>>
>> >>>>
>> >>>> hey mike,
>> >>>>
>> >>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>> >>>>
>> >>>> RangeBeginningDate > '2007-01-01'
>> >>>>
>> >>>> Should be a space before and after >
>> >>>>
>> >>>> each value should start and end with single quote.
>> >>>>
>> >>>> -brian
>> >>>>
>> >>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>
>> >>>>> Hey Brian,
>> >>>>>
>> >>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>> >>>>>
>> >>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>> >>>>>
>> >>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>> >>>>>
>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>> >>>>>
>> >>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>> >>>>>
>> >>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>> >>>>>
>> >>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>> >>>>>
>> >>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>> >>>>>
>> >>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>> >>>>>
>> >>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>> >>>>>
>> >>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>> >>>>>
>> >>>>>
>> >>>>> -Mike
>> >>>>>
>> >>>>> From: Brian Foster <ho...@mac.com>
>> >>>>> Reply-To: <us...@oodt.apache.org>
>> >>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>> >>>>> To: <us...@oodt.apache.org>
>> >>>>> Subject: Re: Query Tool Bugs?
>> >>>>>
>> >>>>>
>> >>>>> hey mike,
>> >>>>>
>> >>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>> >>>>>
>> >>>>> -brian
>> >>>>>
>> >>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>>
>> >>>>>> Hey Brian,
>> >>>>>>
>> >>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>> >>>>>>
>> >>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>> >>>>>>
>> >>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>> >>>>>>
>> >>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>> >>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>> >>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>> >>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> >>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> >>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>> >>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>> >>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>> >>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>> >>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>> >>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>> >>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>>
>> >>>>>> Let me know if you'd like for me to file a bug for this.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Mike
>> >>>>>>
>> >>>>>> From: Brian Foster <ho...@mac.com>
>> >>>>>> Reply-To: <us...@oodt.apache.org>
>> >>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>> >>>>>> To: <us...@oodt.apache.org>
>> >>>>>> Subject: Re: Query Tool Bugs?
>> >>>>>>
>> >>>>>> hey mike,
>> >>>>>>
>> >>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>> >>>>>>
>> >>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>> >>>>>>
>> >>>>>> you have an extra set of double quotes around the where clause... try removing those
>> >>>>>>
>> >>>>>> -brian
>> >>>>>>
>> >>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>> >>>>>>
>> >>>>>>> Hey Chris,
>> >>>>>>>
>> >>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>> >>>>>>> <ch...@jpl.nasa.gov> wrote:
>> >>>>>>>
>> >>>>>>> >Hey Mike,
>> >>>>>>> >
>> >>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>> >>>>>>> >
>> >>>>>>> >>>>
>> >>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>> >>>>>>> >>>>Failed
>> >>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>> >>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>> >>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>> >>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>> >>>>>>> >>>>(X
>> >>>>>>> >>>> mlRpcFileManagerClient.java:958)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>> >>>>>>> >>>>av
>> >>>>>>> >>>> a:251)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>>>>> >>>
>> >>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>> >>>>>>> >>> like a partial one, missing the timezone do you think that might
>> >>>>>>> >>> affect ir?
>> >>>>>>> >>
>> >>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>> >>>>>>> >> time be split when performing a query. Reason being is that the query
>> >>>>>>> >>tool
>> >>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>> >>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>> >>>>>>> >>work
>> >>>>>>> >> for him and the way around it was to split it up. I think behind the
>> >>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>> >>>>>>> >>might
>> >>>>>>> >> be why the tool might be having performance issues?
>> >>>>>>> >
>> >>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>> >>>>>>> >manager,
>> >>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>> >>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>> >>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>> >>>>>>> >because,
>> >>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>> >>>>>>> >to a
>> >>>>>>> >number (which would be the case if you used the
>> >>>>>>> >DataSourceRepositoryManager --
>> >>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>> >>>>>>> >one).
>> >>>>>>>
>> >>>>>>> My repository manager and catalog combination is defined as follows in the
>> >>>>>>> filemgr.properties:
>> >>>>>>>
>> >>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>> >>>>>>> toryManagerFactory
>> >>>>>>>
>> >>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>> >>>>>>> ogFactory
>> >>>>>>>
>> >>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >>
>> >>>>>>> >>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>> >>>>>>> >>>>luck:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>> >>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>> >>>>>>> >>>> Ambiguous output redirect.
>> >>>>>>> >>>>
>> >>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>> >>>>>>> >>>> tool is expecting.
>> >>>>>>> >>>
>> >>>>>>> >>> Did you check the code in SVN?
>> >>>>>>> >>
>> >>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> >>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>> >>>>>>> >>see
>> >>>>>>> >> if I get the same results.
>> >>>>>>> >
>> >>>>>>> >I think there is a fix for something similar to this in the trunk (as
>> >>>>>>> >bfoster mentioned),
>> >>>>>>> >but thinking about this more, I bet you're having the quoteFields
>> >>>>>>> >problem, per
>> >>>>>>> >above. Scope it out and let me know.
>> >>>>>>>
>> >>>>>>> Based on the properties specified above, the quoteFields option is turned
>> >>>>>>> on. So I think the properties are set correctly, no?
>> >>>>>>> It'll be interesting to see if Brian was able to find an error in the
>> >>>>>>> policy files that I sent in the previous e-mail.
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >>
>> >>>>>>> >>>
>> >>>>>>> >>>>
>> >>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>> >>>>>>> >>>> dates. I tried the following query below and got the following output:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>> >>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>> >>>>>>> >>>> for more info.
>> >>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>> >>>>>>> >>>> String index out of range: -1
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>> >>>>>>> >>>>1)
>> >>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>> >>>>>>> >>>>av
>> >>>>>>> >>>> a:255)
>> >>>>>>> >>>> at
>> >>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> >>>>>>> >>>>
>> >>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>> >>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>> >>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>> >>>>>>> >>>> on my query, the tool returns a result as expected:
>> >>>>>>> >>>>
>> >>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>> >>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>> >>>>>>> >>>> log4j:WARN No appenders could be found for logger
>> >>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>> >>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>> >>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>> >>>>>>> >>>> for more info.
>> >>>>>>> >>>>
>> >>>>>>> >>>>
>> >>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>> >>>>>>> >>>>51
>> >>>>>>> >>>>
>> >>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>> >>>>>>> >>>>5-
>> >>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>> >>>>>>> >>>
>> >>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>> >>>>>>> >>>issue?
>> >>>>>>> >>>
>> >>>>>>> >>> Feel free to file a bug and would love you to investigate!
>> >>>>>>> >>
>> >>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>> >>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>> >>>>>>> >> indeed a bug.
>> >>>>>>> >
>> >>>>>>> >Great Mike, thanks.
>> >>>>>>> >
>> >>>>>>> >Cheers,
>> >>>>>>> >Chris
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Mike
>> >>>>>>>
>> >>>>>>> >
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >Chris Mattmann, Ph.D.
>> >>>>>>> >Senior Computer Scientist
>> >>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> >>>>>>> >Office: 171-266B, Mailstop: 171-246
>> >>>>>>> >Email: chris.a.mattmann@nasa.gov
>> >>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >Adjunct Assistant Professor, Computer Science Department
>> >>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>> >>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> >>>>>>> >
>> >>>>>>>
>> >>>> #!/bin/sh
>> >>>>
>> >>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>> > <createProductTypeSQL.sh>
>>
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey chris,
this is an SQL compatiblity thing... MySQL doesn't support INTERSECT... i created an issue for this fix:�(OODT-460) DataSourceCatalog doesn't work with MySQL
-brian
On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
This is where quoteFields comes in.
If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
the element IDs being numeric, they are strings.
That's Brian is talking about I think. I you mentioned you were using quoteFields so
it should take care of this, but seeing the SQL you're generating can help.
Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
Cheers,
Chris
On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>
> -Mike
>
> From: Brian Foster <ho...@mac.com>
> Reply-To: <us...@oodt.apache.org>
> Date: Mon, 21 May 2012 22:32:30 +0000
> To: <us...@oodt.apache.org>
> Subject: Re: Query Tool Bugs?
>
>
> hey mike,
>
> i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>
> -brian
>
> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian,
>>
>> Hmmm...interesting.
>>
>> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>>
>> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>>
>> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>>
>> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>>
>> Thanks again for all your help on this,
>> Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 19:12:09 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>>
>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>
>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> INFO: Query returned 0 results
>>
>> There must be something special about your setup...
>>
>> -brian
>>
>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>
>>>
>>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>>>
>>> i ran the following:
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>>
>>> I got the following file manager query:
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>>>
>>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>>>
>>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>
>>>> Hey Brian,
>>>>
>>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>>>>
>>>> The exception I'm getting is this:
>>>>
>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> ... 2 more
>>>>
>>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>>>>
>>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>>>>
>>>> Thanks for your help!
>>>>
>>>> -Mike
>>>>
>>>> From: Brian Foster <ho...@mac.com>
>>>> Reply-To: <us...@oodt.apache.org>
>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>> To: <us...@oodt.apache.org>
>>>> Subject: Re: Query Tool Bugs?
>>>>
>>>>
>>>> hey mike,
>>>>
>>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>>>>
>>>> RangeBeginningDate > '2007-01-01'
>>>>
>>>> Should be a space before and after >
>>>>
>>>> each value should start and end with single quote.
>>>>
>>>> -brian
>>>>
>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>>>>>
>>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>>>>>
>>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>>>>>
>>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>>>>>
>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>>>>>
>>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>>>>>
>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>
>>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>>>>>
>>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>>>>>>
>>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>>>>>>
>>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>>>>>>
>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>
>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>>>>>>
>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>>>>>>
>>>>>> you have an extra set of double quotes around the where clause... try removing those
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Chris,
>>>>>>>
>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>> >Hey Mike,
>>>>>>> >
>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>> >
>>>>>>> >>>>
>>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>> >>>>Failed
>>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>>>> >>>>(X
>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:251)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>
>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>>>>>> >>> like a partial one, missing the timezone do you think that might
>>>>>>> >>> affect ir?
>>>>>>> >>
>>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>>>>>>> >> time be split when performing a query. Reason being is that the query
>>>>>>> >>tool
>>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>>>>>>> >>work
>>>>>>> >> for him and the way around it was to split it up. I think behind the
>>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>>>>>>> >>might
>>>>>>> >> be why the tool might be having performance issues?
>>>>>>> >
>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>>>>>>> >manager,
>>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>>>>>>> >because,
>>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>>>>>>> >to a
>>>>>>> >number (which would be the case if you used the
>>>>>>> >DataSourceRepositoryManager --
>>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>>>>>>> >one).
>>>>>>>
>>>>>>> My repository manager and catalog combination is defined as follows in the
>>>>>>> filemgr.properties:
>>>>>>>
>>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>>>>>>> toryManagerFactory
>>>>>>>
>>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>>>>>>> ogFactory
>>>>>>>
>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>>>>>>> >>>>luck:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>> >>>> Ambiguous output redirect.
>>>>>>> >>>>
>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>>>>>>> >>>> tool is expecting.
>>>>>>> >>>
>>>>>>> >>> Did you check the code in SVN?
>>>>>>> >>
>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>>>>>>> >>see
>>>>>>> >> if I get the same results.
>>>>>>> >
>>>>>>> >I think there is a fix for something similar to this in the trunk (as
>>>>>>> >bfoster mentioned),
>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>> >problem, per
>>>>>>> >above. Scope it out and let me know.
>>>>>>>
>>>>>>> Based on the properties specified above, the quoteFields option is turned
>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>> It'll be interesting to see if Brian was able to find an error in the
>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>>>>>>> >>>> dates. I tried the following query below and got the following output:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>>>>> >>>> String index out of range: -1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>>>> >>>>1)
>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:255)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>>
>>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>>>> >>>>51
>>>>>>> >>>>
>>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>>>> >>>>5-
>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>> >>>
>>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>>>>> >>>issue?
>>>>>>> >>>
>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>> >>
>>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>>>>>>> >> indeed a bug.
>>>>>>> >
>>>>>>> >Great Mike, thanks.
>>>>>>> >
>>>>>>> >Cheers,
>>>>>>> >Chris
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> >
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>> >Senior Computer Scientist
>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >
>>>>>>>
>>>> #!/bin/sh
>>>>
>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
> <createProductTypeSQL.sh>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Chris/Brian,
Here are my CREATE SQL statements to create the tables in the mysql db:
I'm using a shell script to create tables for the various product types
that the File Manager will accept:
export product_types=\
"SPICE_TRAJ SPICE_ATT SPICE_ANT SPICE_EARTH SPICE_SCLK SPICE_FRAMES
SPICE_LEAP
SPICE_PCK SPICE_EFF SPICE_PTK \
InputConfig OutputConfig MetConfig Parameter \
SCAN NCDC_GSOD_WMO \
RunConfig Log QA Monitor\
RadarProductType RadiometerProductType L2_SM_ProductType L3_SM_ProductType
\
L0a_Radar L0a_Radiometer L0b_Radar
L1A_Radar L1B_S0_LoRes L1C_S0_HiRes L0b_Radiometer \
L1A_Radiometer L1B_TB L1C_TB L2_SM_A L2_SM_P L2_SM_AP \
L3_SM_A L3_SM_P L3_SM_AP L3_FT_A \
GenericFile SPDM_Type TimeBasedProductType GenericProductType \
DownlinkBasedProductType SMAP_CoreProductType HalfOrbitBasedProductType
DailyBasedProductType"
for i in ${product_types}
do
echo "CREATE TABLE ${i}_metadata"
echo "("
echo " product_id int NOT NULL,"
echo " element_id varchar(1000) NOT NULL,"
echo " metadata_value varchar(2500) NOT NULL"
echo ");"
echo "CREATE TABLE ${i}_reference"
echo "("
echo " product_id int NOT NULL,"
echo " product_orig_reference varchar(2000) NOT NULL,"
echo " product_datastore_reference varchar(2000),"
echo " product_reference_filesize int NOT NULL,"
echo " product_reference_mimetype varchar(50)"
echo ");"
done
CREATE TABLE `products` (
`product_id` int(11) NOT NULL auto_increment,
`product_structure` varchar(20) NOT NULL default '',
`product_type_id` varchar(255) NOT NULL default '',
`product_name` varchar(255) NOT NULL default '',
`product_transfer_status` varchar(255) NOT NULL default 'TRANSFERING',
PRIMARY KEY (`product_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
Here is what the log is showing when running the following query tool
command:
%> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime >
'12:00:00.000'"
FINE: catalog query: executing: (SELECT DISTINCT product_id FROM
L0a_Radar_metadata WHERE element_id =
'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000')
INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE
element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value >
'2007-01-01') ORDER BY product_id DESC
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an
error in your SQL syntax; check the manual that corresponds to your MySQL
server version for the right syntax to use near 'INTERSECT (SELECT
DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcce
ssorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstru
ctorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
at com.mysql.jdbc.Util.getInstance(Util.java:381)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1030)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3558)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3490)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1959)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2109)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2637)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2566)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1464)
at
org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatemen
t.java:205)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.paginateQuery(DataSou
rceCatalog.java:1965)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.query(DataSourceCatal
og.java:1179)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFil
eManager.java:600)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:3
9)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.xmlrpc.Invoker.execute(Invoker.java:130)
at org.apache.xmlrpc.XmlRpcWorker.invokeHandler(XmlRpcWorker.java:84)
at org.apache.xmlrpc.XmlRpcWorker.execute(XmlRpcWorker.java:146)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:139)
at org.apache.xmlrpc.XmlRpcServer.execute(XmlRpcServer.java:125)
at org.apache.xmlrpc.WebServer$Connection.run(WebServer.java:761)
at org.apache.xmlrpc.WebServer$Runner.run(WebServer.java:642)
at java.lang.Thread.run(Thread.java:680)
May 23, 2012 8:39:58 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
WARNING: Exception performing query. Message: You have an error in your
SQL syntax; check the manual that corresponds to your MySQL server version
for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id
FROM L0a_Radar_metadata WHERE element_id =' at line 1
May 23, 2012 8:39:58 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
SEVERE: Unable to rollback query transaction. Message: Can't call rollback
when autocommit=true
org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: You have
an error in your SQL syntax; check the manual that corresponds to your
MySQL server version for the right syntax to use near 'INTERSECT (SELECT
DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.paginateQuery(DataSou
rceCatalog.java:2020)
at
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog.query(DataSourceCatal
og.java:1179)
at
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager.complexQuery(XmlRpcFil
eManager.java:600)
I did another quick test where I used an 'OR' instead of an 'AND' in my
query and I got a result back:
%> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
L0a_Radar WHERE RangeBeginningDate > '2007-01-01' OR RangeBeginningTime >
'12:00:00.000'"
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
FINE: catalog query: executing: (SELECT DISTINCT product_id FROM
L0a_Radar_metadata WHERE element_id =
'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000')
UNION (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id
= 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01')
ORDER BY product_id DESC
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog getProductById
FINE: getProductById: Executing: SELECT * FROM products WHERE product_id =
314
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog getMetadata
FINE: getMetadata: Executing: SELECT * FROM L0a_Radar_metadata WHERE
product_id = 314
May 23, 2012 8:51:14 AM
org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 1 results
I'm starting to convince myself that the issue here is that query tool
converts a user-given query with an 'AND' in a WHERE clause into a query
containing an INTERSECT syntax, which isn't supported in mysql. That's the
reason that an exception gets thrown.
I don't know if there is a way around this by configuring the file manager
differently (using something else other than XMLValidationLayer and/or
DataSourceCatalog).
As I've mentioned previously, SMAP will be moving to an oracle db very
soon, so this won't be an issue any longer as the INTERSECT syntax is
supported there. But this is an interesting finding and it's a lesson
learned. :)
-Mike
On 5/22/12 7:55 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>This is where quoteFields comes in.
>
>If you are using a combination of XMLValidationLayer with
>DataSourceCatalog, rather than
>the element IDs being numeric, they are strings.
>
>That's Brian is talking about I think. I you mentioned you were using
>quoteFields so
>it should take care of this, but seeing the SQL you're generating can
>help.
>
>Can you edit etc/logging.properties and turn the catalog level WAY up
>like to ALL?
>
>Then restart FM and tell us the queries that are printing from the
>logs/cas_filemgrX.log.Y.
>
>Cheers,
>Chris
>
>On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
>
>> Hey Brian,
>>
>> element_id is set to a varchar. I'm using the attached shell script to
>>create the database tables. Since I'm using XMLValidationLayer, does
>>this mean that I should be creating a different set of tables? If so,
>>what would those be? I'm assuming the provided
>>cas-filemgr-schema-mysql.sql is used only if you're using
>>DataSourceValidationLayer...
>>
>> -Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 22:32:30 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike,
>>
>> i copied in your etc files and policy files into a newly deployed trunk
>>filemgr deployment... i then just commented out the code which talked to
>>sql driver so i could see the SQL which would be sent to the sql
>>driver... basically by doing that i've narrowed it down to something
>>with your SQL connection... did you change element_id to a varchar
>>instead of and int since you are using XMLValidationLayer... the provide
>>cas-filemgr-schema-mysql.sql is setup to work with
>>DataSourceValidationLayer... i think you are getting an error because
>>there are quotes around your element_id and the database is expecting a
>>integer (the quotes are correct in this case though, you DB is just
>>configured wrong)
>>
>> -brian
>>
>> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)"
>><mi...@jpl.nasa.gov> wrote:
>>
>>> Hey Brian,
>>>
>>> Hmmm...interesting.
>>>
>>> I believe I attached a filemgr.properties in an earlier e-mail. Did
>>>you use this in your configuration as well?
>>>
>>> Also, which version of the filemgr are you using on your computer? The
>>>trunk? Or the wengine-branch that you pointed me to earlier?
>>>
>>> I'm also using a mysql DB with the file manager. I don't think that
>>>should matter though in this case, but I could be wrong. I'm assuming
>>>you set up your file manager with some other db backend?
>>>
>>> Would it be possible to send me a tar package of your file manger
>>>deployment that you have on your computer? I want to see if I can just
>>>untar it on my computer, start it up and see if that works.
>>>
>>> Thanks again for all your help on this,
>>> Mike
>>>
>>> From: Brian Foster <ho...@mac.com>
>>> Reply-To: <us...@oodt.apache.org>
>>> Date: Mon, 21 May 2012 19:12:09 +0000
>>> To: <us...@oodt.apache.org>
>>> Subject: Re: Query Tool Bugs?
>>>
>>>
>>> hey mike... i'm kinda stumped... used your configuration on a filemgr
>>>on my computer and i get the following output:
>>>
>>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>RangeBeginningTime > '12:00:00.000'"
>>>
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>RangeBeginningDate:[2007-01-01 TO null])
>>> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningTime' AND metadata_value >
>>>'12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =
>>>'urn:smap:product:RangeBeginningDate' AND metadata_value >
>>>'2007-01-01') ORDER BY product_id DESC
>>> May 21, 2012 12:09:58 PM
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>>> INFO: Query returned 0 results
>>>
>>> There must be something special about your setup...
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>>
>>>>
>>>> Hey mike... i should have some time today to take a look into this
>>>>more than just throwing out guesses... i put a print statement in the
>>>>QueryTool code to see what the SqlParser was doing with the query...
>>>>
>>>> i ran the following:
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND
>>>>RangeBeginningTime > '12:00:00.000'"
>>>>
>>>> I got the following file manager query:
>>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null],
>>>>RangeBeginningDate:[2007-01-01 TO null])
>>>>
>>>> so it appears that it doesn't have anything to do with shell parsing
>>>>(like i initially thought) or SqlParser... i will take a closer look
>>>>at your configuration today and let you know what i find.
>>>>
>>>> also something that has been know to cause problems in the past is
>>>>shell aliases... maybe take a look at those if you have any... in the
>>>>mean time i will try and get to the bottom of this.
>>>>
>>>> -brian
>>>>
>>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)"
>>>><mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> Yep, I tried the query with spaces before and after > and also
>>>>>surrounded each value with a single quote and it didn't work for me.
>>>>>I also put the query into the shell script as you've suggested and
>>>>>still no luck.
>>>>>
>>>>> The exception I'm getting is this:
>>>>>
>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpc
>>>>>ClientResponseProcessor.java:104)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcC
>>>>>lientResponseProcessor.java:71)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:7
>>>>>3)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:945)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.
>>>>>java:250)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>> Exception in thread "main"
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:951)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.
>>>>>java:250)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>version for the right syntax to use near 'INTERSECT (SELECT DISTINCT
>>>>>product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpc
>>>>>ClientResponseProcessor.java:104)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcC
>>>>>lientResponseProcessor.java:71)
>>>>> at
>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:7
>>>>>3)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>> at
>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuer
>>>>>y(XmlRpcFileManagerClient.java:945)
>>>>> ... 2 more
>>>>>
>>>>> Attached is the query tool script, which contains the query I'm
>>>>>trying. It appears that this is a bug. Since query tool has been
>>>>>deprecated in the trunk in favor of the super new file manager client
>>>>>tool, then I'll go ahead and file a JIRA issue for this against the
>>>>>file manager.
>>>>>
>>>>> In the meantime, I'll definitely look more into this and see if I
>>>>>can find out where the bug is at.
>>>>>
>>>>> Thanks for your help!
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> You putting a space between the operators? its hard to tell in
>>>>>emails, stuff gets normalized sometimes.
>>>>>
>>>>> RangeBeginningDate > '2007-01-01'
>>>>>
>>>>> Should be a space before and after >
>>>>>
>>>>> each value should start and end with single quote.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)"
>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> I checked out the wengine-branch filemgr and unfortunately, it
>>>>>>still didn't work. I tried a query using the wengine branch's
>>>>>>query_tool since this version of the filemgr-client didn't have the
>>>>>>ability to query.
>>>>>>
>>>>>> In any case, I took the liberty of diving into the Query Tool code
>>>>>>to see if I can find out more as to what is going on. So this is
>>>>>>what I found:
>>>>>>
>>>>>> Surrounding single quotes around each of the conditions got me
>>>>>>further into the code. So, when I entered the query as follows,
>>>>>>
>>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND
>>>>>>'RangeBeginningTime>12:00:00Z'
>>>>>>
>>>>>> The SQLParser class liked the syntax and it produced the following
>>>>>>post fix statement:
>>>>>>
>>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01',
>>>>>>'RangeBeginningTime>12:00:00Z', AND]
>>>>>>
>>>>>> I think from here, it creates some Boolean Query Criteria with the
>>>>>>following content:
>>>>>>
>>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO
>>>>>>null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>>
>>>>>> I noticed that there are unmatched single quotes around the date
>>>>>>and time values. Not sure how that came about. The tool still threw
>>>>>>an exception. I'm guessing it's because of the unmatched single
>>>>>>quotes.
>>>>>>
>>>>>> I tried surrounding the date and time values with single quotes as
>>>>>>well, but the SQLParser class didn't like that either:
>>>>>>
>>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND
>>>>>>'RangeBeginningTime>'12:00:00Z''
>>>>>>
>>>>>>
>>>>>> -Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> would you mind trying to run this query using the wengine-branch
>>>>>>filemgr
>>>>>>(https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemg
>>>>>>r/)... there are a lot of bug fixes floating around in both Lucene
>>>>>>and DataSource Catalogs in that branch... if it works with that then
>>>>>>we know it is a bug with trunk.
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)"
>>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Brian,
>>>>>>>
>>>>>>> Ya, I tried the query with and without the double quotes
>>>>>>>surrounding the where clause and they both didn't work for me.
>>>>>>>
>>>>>>> In any case, I've built the latest file manager in the trunk and
>>>>>>>it fixed that StringIndexOutofBoundsException issue I was seeing.
>>>>>>>Thanks!
>>>>>>>
>>>>>>> Out of curiosity, I tried using the latest File Manager Client
>>>>>>>tool with the query with multiple conditions to see if I could get
>>>>>>>a result, but it threw an exception as well:
>>>>>>>
>>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM
>>>>>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>>>>>RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>>> May 16, 2012 7:20:53 AM
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>>> INFO: Loading File Manager Configuration Properties from:
>>>>>>>[../etc/filemgr.properties]
>>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>>>version for the right syntax to use near 'INTERSECT (SELECT
>>>>>>>DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at
>>>>>>>line 1
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlR
>>>>>>>pcClientResponseProcessor.java:104)
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRp
>>>>>>>cClientResponseProcessor.java:71)
>>>>>>> at
>>>>>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java
>>>>>>>:73)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQu
>>>>>>>ery(XmlRpcFileManagerClient.java:990)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execut
>>>>>>>e(AbstractQueryCliAction.java:75)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:3
>>>>>>>31)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>>> at
>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlR
>>>>>>>pcFileManagerClient.java:1323)
>>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat
>>>>>>>'null', and delimiter '
>>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null',
>>>>>>>endDateTimeMetKey 'null', priorityMetKey 'null', null' :
>>>>>>>java.lang.Exception:
>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>Failed to perform complex query : You have an error in your SQL
>>>>>>>syntax; check the manual that corresponds to your MySQL server
>>>>>>>version for the right syntax to use near 'INTERSECT (SELECT
>>>>>>>DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at
>>>>>>>line 1
>>>>>>>
>>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>>> To: <us...@oodt.apache.org>
>>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>>
>>>>>>> hey mike,
>>>>>>>
>>>>>>> first pass over the policy files look good... however, is the way
>>>>>>>you posted the query the way you are running it?
>>>>>>>
>>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT *
>>>>>>>>FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>>>RangeBeginningTime>'12:00:00.000Z'""
>>>>>>>
>>>>>>> you have an extra set of double quotes around the where clause...
>>>>>>>try removing those
>>>>>>>
>>>>>>> -brian
>>>>>>>
>>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)"
>>>>>>><mi...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>>> Hey Chris,
>>>>>>>>
>>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>>
>>>>>>>> >Hey Mike,
>>>>>>>> >
>>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>>> >
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>>> >>>>Failed
>>>>>>>> >>>> to perform complex query : You have an error in your SQL
>>>>>>>>syntax; check
>>>>>>>> >>>> the manual that corresponds to your MySQL server version for
>>>>>>>>the right
>>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id
>>>>>>>>FROM
>>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.comp
>>>>>>>>>>>>lexQuery
>>>>>>>> >>>>(X
>>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(Que
>>>>>>>>>>>>ryTool.j
>>>>>>>> >>>>av
>>>>>>>> >>>> a:251)
>>>>>>>> >>>> at
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java
>>>>>>>>>>>>:241)
>>>>>>>> >>>
>>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time
>>>>>>>>format? Looks
>>>>>>>> >>> like a partial one, missing the timezone do you think that
>>>>>>>>might
>>>>>>>> >>> affect ir?
>>>>>>>> >>
>>>>>>>> >> I talked with Rishi regarding this and he recommended that the
>>>>>>>>date and
>>>>>>>> >> time be split when performing a query. Reason being is that
>>>>>>>>the query
>>>>>>>> >>tool
>>>>>>>> >> blows up when trying to compare datetime values. He mentioned
>>>>>>>>that he
>>>>>>>> >> tried querying against ISO 8601 date/time values before and it
>>>>>>>>didn't
>>>>>>>> >>work
>>>>>>>> >> for him and the way around it was to split it up. I think
>>>>>>>>behind the
>>>>>>>> >> scenes, the query tool is actually doing an ascii comparison,
>>>>>>>>which
>>>>>>>> >>might
>>>>>>>> >> be why the tool might be having performance issues?
>>>>>>>> >
>>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your
>>>>>>>>repository
>>>>>>>> >manager,
>>>>>>>> >and catalog combination? If you are using e.g., a
>>>>>>>>DataSourceCatalog,
>>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the
>>>>>>>>quoteFields
>>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This
>>>>>>>>is
>>>>>>>> >because,
>>>>>>>> >in these scenarios, the identifier for elementIds is a string,
>>>>>>>>compared
>>>>>>>> >to a
>>>>>>>> >number (which would be the case if you used the
>>>>>>>> >DataSourceRepositoryManager --
>>>>>>>> >the short answer there is don't, it's not as well maintained as
>>>>>>>>the XML
>>>>>>>> >one).
>>>>>>>>
>>>>>>>> My repository manager and catalog combination is defined as
>>>>>>>>follows in the
>>>>>>>> filemgr.properties:
>>>>>>>>
>>>>>>>>
>>>>>>>>filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.X
>>>>>>>>MLReposi
>>>>>>>> toryManagerFactory
>>>>>>>>
>>>>>>>>
>>>>>>>>filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSou
>>>>>>>>rceCatal
>>>>>>>> ogFactory
>>>>>>>>
>>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> >
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>>>
>>>>>>>> >>>> I tried surrounding the entire condition with quotes, but
>>>>>>>>still no
>>>>>>>> >>>>luck:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>>> >>>> Ambiguous output redirect.
>>>>>>>> >>>>
>>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know
>>>>>>>>what the
>>>>>>>> >>>> tool is expecting.
>>>>>>>> >>>
>>>>>>>> >>> Did you check the code in SVN?
>>>>>>>> >>
>>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have
>>>>>>>>the code
>>>>>>>> >> checked out onto my local machine. I can certainly build the
>>>>>>>>trunk and
>>>>>>>> >>see
>>>>>>>> >> if I get the same results.
>>>>>>>> >
>>>>>>>> >I think there is a fix for something similar to this in the
>>>>>>>>trunk (as
>>>>>>>> >bfoster mentioned),
>>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>>> >problem, per
>>>>>>>> >above. Scope it out and let me know.
>>>>>>>>
>>>>>>>> Based on the properties specified above, the quoteFields option
>>>>>>>>is turned
>>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>>> It'll be interesting to see if Brian was able to find an error in
>>>>>>>>the
>>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>>
>>>>>>>> >
>>>>>>>> >>
>>>>>>>> >>>
>>>>>>>> >>>>
>>>>>>>> >>>> My second issue that I'm running into is in regards to
>>>>>>>>querying of
>>>>>>>> >>>> dates. I tried the following query below and got the
>>>>>>>>following output:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>>> >>>> log4j:WARN See
>>>>>>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>>> >>>> for more info.
>>>>>>>> >>>> Exception in thread "main"
>>>>>>>>java.lang.StringIndexOutOfBoundsException:
>>>>>>>> >>>> String index out of range: -1
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder
>>>>>>>>>>>>.java:88
>>>>>>>> >>>>1)
>>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>>> >>>> at
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(Que
>>>>>>>>>>>>ryTool.j
>>>>>>>> >>>>av
>>>>>>>> >>>> a:255)
>>>>>>>> >>>> at
>>>>>>>>
>>>>>>>>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java
>>>>>>>>>>>>:241)
>>>>>>>> >>>>
>>>>>>>> >>>> For this particular product, I have 1 product in my catalog
>>>>>>>>where the
>>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if
>>>>>>>>that factors
>>>>>>>> >>>> into why an exception is being thrown here. When I use an
>>>>>>>>earlier date
>>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>>> >>>>
>>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query
>>>>>>>>"SELECT * FROM
>>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>>> >>>> log4j:WARN See
>>>>>>>>http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>>> >>>> for more info.
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_R
>>>>>>>>>>>>adar,V20
>>>>>>>> >>>>51
>>>>>>>> >>>>
>>>>>>>>
>>>>>>>>>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:
>>>>>>>>>>>>27:59.38
>>>>>>>> >>>>5-
>>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>>> >>>
>>>>>>>> >>> Interesting! Did you scope the code to see if there's a
>>>>>>>>RangeQuery
>>>>>>>> >>>issue?
>>>>>>>> >>>
>>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>>> >>
>>>>>>>> >> I haven't dived into the code, but will certainly do this as
>>>>>>>>SMAP will
>>>>>>>> >> need these capabilities. I will file a bug if it turns out
>>>>>>>>that this is
>>>>>>>> >> indeed a bug.
>>>>>>>> >
>>>>>>>> >Great Mike, thanks.
>>>>>>>> >
>>>>>>>> >Cheers,
>>>>>>>> >Chris
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Mike
>>>>>>>>
>>>>>>>> >
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>>> >Senior Computer Scientist
>>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>>>
>>>>>>>>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>> >
>>>>>>>>
>>>>> #!/bin/sh
>>>>>
>>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib
>>>>>org.apache.oodt.cas.filemgr.tools.QueryTool --url
>>>>>http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE
>>>>>RangeBeginningDate > '2007-01-01' AND RangeBeginningTime >
>>>>>'12:00:00.000'"
>> <createProductTypeSQL.sh>
>
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey chris,
this is an SQL compatiblity thing... MySQL doesn't support INTERSECT... i created an issue for this fix: (OODT-460) DataSourceCatalog doesn't work with MySQL
-brian
On May 22, 2012, at 07:55 PM, "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov> wrote:
Hi Mike,
This is where quoteFields comes in.
If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
the element IDs being numeric, they are strings.
That's Brian is talking about I think. I you mentioned you were using quoteFields so
it should take care of this, but seeing the SQL you're generating can help.
Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
Cheers,
Chris
On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>
> -Mike
>
> From: Brian Foster <ho...@mac.com>
> Reply-To: <us...@oodt.apache.org>
> Date: Mon, 21 May 2012 22:32:30 +0000
> To: <us...@oodt.apache.org>
> Subject: Re: Query Tool Bugs?
>
>
> hey mike,
>
> i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>
> -brian
>
> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian,
>>
>> Hmmm...interesting.
>>
>> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>>
>> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>>
>> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>>
>> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>>
>> Thanks again for all your help on this,
>> Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 19:12:09 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>>
>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>
>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> INFO: Query returned 0 results
>>
>> There must be something special about your setup...
>>
>> -brian
>>
>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>
>>>
>>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>>>
>>> i ran the following:
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>>
>>> I got the following file manager query:
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>>>
>>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>>>
>>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>
>>>> Hey Brian,
>>>>
>>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>>>>
>>>> The exception I'm getting is this:
>>>>
>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> ... 2 more
>>>>
>>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>>>>
>>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>>>>
>>>> Thanks for your help!
>>>>
>>>> -Mike
>>>>
>>>> From: Brian Foster <ho...@mac.com>
>>>> Reply-To: <us...@oodt.apache.org>
>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>> To: <us...@oodt.apache.org>
>>>> Subject: Re: Query Tool Bugs?
>>>>
>>>>
>>>> hey mike,
>>>>
>>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>>>>
>>>> RangeBeginningDate > '2007-01-01'
>>>>
>>>> Should be a space before and after >
>>>>
>>>> each value should start and end with single quote.
>>>>
>>>> -brian
>>>>
>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>>>>>
>>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>>>>>
>>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>>>>>
>>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>>>>>
>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>>>>>
>>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>>>>>
>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>
>>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>>>>>
>>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>>>>>>
>>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>>>>>>
>>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>>>>>>
>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>
>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>>>>>>
>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>>>>>>
>>>>>> you have an extra set of double quotes around the where clause... try removing those
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Chris,
>>>>>>>
>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>> >Hey Mike,
>>>>>>> >
>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>> >
>>>>>>> >>>>
>>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>> >>>>Failed
>>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>>>> >>>>(X
>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:251)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>
>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>>>>>> >>> like a partial one, missing the timezone do you think that might
>>>>>>> >>> affect ir?
>>>>>>> >>
>>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>>>>>>> >> time be split when performing a query. Reason being is that the query
>>>>>>> >>tool
>>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>>>>>>> >>work
>>>>>>> >> for him and the way around it was to split it up. I think behind the
>>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>>>>>>> >>might
>>>>>>> >> be why the tool might be having performance issues?
>>>>>>> >
>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>>>>>>> >manager,
>>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>>>>>>> >because,
>>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>>>>>>> >to a
>>>>>>> >number (which would be the case if you used the
>>>>>>> >DataSourceRepositoryManager --
>>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>>>>>>> >one).
>>>>>>>
>>>>>>> My repository manager and catalog combination is defined as follows in the
>>>>>>> filemgr.properties:
>>>>>>>
>>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>>>>>>> toryManagerFactory
>>>>>>>
>>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>>>>>>> ogFactory
>>>>>>>
>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>>>>>>> >>>>luck:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>> >>>> Ambiguous output redirect.
>>>>>>> >>>>
>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>>>>>>> >>>> tool is expecting.
>>>>>>> >>>
>>>>>>> >>> Did you check the code in SVN?
>>>>>>> >>
>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>>>>>>> >>see
>>>>>>> >> if I get the same results.
>>>>>>> >
>>>>>>> >I think there is a fix for something similar to this in the trunk (as
>>>>>>> >bfoster mentioned),
>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>> >problem, per
>>>>>>> >above. Scope it out and let me know.
>>>>>>>
>>>>>>> Based on the properties specified above, the quoteFields option is turned
>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>> It'll be interesting to see if Brian was able to find an error in the
>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>>>>>>> >>>> dates. I tried the following query below and got the following output:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>>>>> >>>> String index out of range: -1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>>>> >>>>1)
>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:255)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>>
>>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>>>> >>>>51
>>>>>>> >>>>
>>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>>>> >>>>5-
>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>> >>>
>>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>>>>> >>>issue?
>>>>>>> >>>
>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>> >>
>>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>>>>>>> >> indeed a bug.
>>>>>>> >
>>>>>>> >Great Mike, thanks.
>>>>>>> >
>>>>>>> >Cheers,
>>>>>>> >Chris
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> >
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>> >Senior Computer Scientist
>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >
>>>>>>>
>>>> #!/bin/sh
>>>>
>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
> <createProductTypeSQL.sh>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
This is where quoteFields comes in.
If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
the element IDs being numeric, they are strings.
That's Brian is talking about I think. I you mentioned you were using quoteFields so
it should take care of this, but seeing the SQL you're generating can help.
Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
Cheers,
Chris
On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>
> -Mike
>
> From: Brian Foster <ho...@mac.com>
> Reply-To: <us...@oodt.apache.org>
> Date: Mon, 21 May 2012 22:32:30 +0000
> To: <us...@oodt.apache.org>
> Subject: Re: Query Tool Bugs?
>
>
> hey mike,
>
> i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>
> -brian
>
> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian,
>>
>> Hmmm...interesting.
>>
>> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>>
>> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>>
>> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>>
>> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>>
>> Thanks again for all your help on this,
>> Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 19:12:09 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>>
>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>
>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> INFO: Query returned 0 results
>>
>> There must be something special about your setup...
>>
>> -brian
>>
>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>
>>>
>>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>>>
>>> i ran the following:
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>>
>>> I got the following file manager query:
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>>>
>>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>>>
>>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>
>>>> Hey Brian,
>>>>
>>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>>>>
>>>> The exception I'm getting is this:
>>>>
>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> ... 2 more
>>>>
>>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>>>>
>>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>>>>
>>>> Thanks for your help!
>>>>
>>>> -Mike
>>>>
>>>> From: Brian Foster <ho...@mac.com>
>>>> Reply-To: <us...@oodt.apache.org>
>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>> To: <us...@oodt.apache.org>
>>>> Subject: Re: Query Tool Bugs?
>>>>
>>>>
>>>> hey mike,
>>>>
>>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>>>>
>>>> RangeBeginningDate > '2007-01-01'
>>>>
>>>> Should be a space before and after >
>>>>
>>>> each value should start and end with single quote.
>>>>
>>>> -brian
>>>>
>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>>>>>
>>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>>>>>
>>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>>>>>
>>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>>>>>
>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>>>>>
>>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>>>>>
>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>
>>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>>>>>
>>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>>>>>>
>>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>>>>>>
>>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>>>>>>
>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>
>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>>>>>>
>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>>>>>>
>>>>>> you have an extra set of double quotes around the where clause... try removing those
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Chris,
>>>>>>>
>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>> >Hey Mike,
>>>>>>> >
>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>> >
>>>>>>> >>>>
>>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>> >>>>Failed
>>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>>>> >>>>(X
>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:251)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>
>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>>>>>> >>> like a partial one, missing the timezone do you think that might
>>>>>>> >>> affect ir?
>>>>>>> >>
>>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>>>>>>> >> time be split when performing a query. Reason being is that the query
>>>>>>> >>tool
>>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>>>>>>> >>work
>>>>>>> >> for him and the way around it was to split it up. I think behind the
>>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>>>>>>> >>might
>>>>>>> >> be why the tool might be having performance issues?
>>>>>>> >
>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>>>>>>> >manager,
>>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>>>>>>> >because,
>>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>>>>>>> >to a
>>>>>>> >number (which would be the case if you used the
>>>>>>> >DataSourceRepositoryManager --
>>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>>>>>>> >one).
>>>>>>>
>>>>>>> My repository manager and catalog combination is defined as follows in the
>>>>>>> filemgr.properties:
>>>>>>>
>>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>>>>>>> toryManagerFactory
>>>>>>>
>>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>>>>>>> ogFactory
>>>>>>>
>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>>>>>>> >>>>luck:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>> >>>> Ambiguous output redirect.
>>>>>>> >>>>
>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>>>>>>> >>>> tool is expecting.
>>>>>>> >>>
>>>>>>> >>> Did you check the code in SVN?
>>>>>>> >>
>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>>>>>>> >>see
>>>>>>> >> if I get the same results.
>>>>>>> >
>>>>>>> >I think there is a fix for something similar to this in the trunk (as
>>>>>>> >bfoster mentioned),
>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>> >problem, per
>>>>>>> >above. Scope it out and let me know.
>>>>>>>
>>>>>>> Based on the properties specified above, the quoteFields option is turned
>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>> It'll be interesting to see if Brian was able to find an error in the
>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>>>>>>> >>>> dates. I tried the following query below and got the following output:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>>>>> >>>> String index out of range: -1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>>>> >>>>1)
>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:255)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>>
>>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>>>> >>>>51
>>>>>>> >>>>
>>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>>>> >>>>5-
>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>> >>>
>>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>>>>> >>>issue?
>>>>>>> >>>
>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>> >>
>>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>>>>>>> >> indeed a bug.
>>>>>>> >
>>>>>>> >Great Mike, thanks.
>>>>>>> >
>>>>>>> >Cheers,
>>>>>>> >Chris
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> >
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>> >Senior Computer Scientist
>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >
>>>>>>>
>>>> #!/bin/sh
>>>>
>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
> <createProductTypeSQL.sh>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
This is where quoteFields comes in.
If you are using a combination of XMLValidationLayer with DataSourceCatalog, rather than
the element IDs being numeric, they are strings.
That's Brian is talking about I think. I you mentioned you were using quoteFields so
it should take care of this, but seeing the SQL you're generating can help.
Can you edit etc/logging.properties and turn the catalog level WAY up like to ALL?
Then restart FM and tell us the queries that are printing from the logs/cas_filemgrX.log.Y.
Cheers,
Chris
On May 22, 2012, at 8:32 AM, Cayanan, Michael D (388J) wrote:
> Hey Brian,
>
> element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
>
> -Mike
>
> From: Brian Foster <ho...@mac.com>
> Reply-To: <us...@oodt.apache.org>
> Date: Mon, 21 May 2012 22:32:30 +0000
> To: <us...@oodt.apache.org>
> Subject: Re: Query Tool Bugs?
>
>
> hey mike,
>
> i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
>
> -brian
>
> On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>
>> Hey Brian,
>>
>> Hmmm...interesting.
>>
>> I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
>>
>> Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
>>
>> I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
>>
>> Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
>>
>> Thanks again for all your help on this,
>> Mike
>>
>> From: Brian Foster <ho...@mac.com>
>> Reply-To: <us...@oodt.apache.org>
>> Date: Mon, 21 May 2012 19:12:09 +0000
>> To: <us...@oodt.apache.org>
>> Subject: Re: Query Tool Bugs?
>>
>>
>> hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
>>
>> $ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>
>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
>> INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
>> May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
>> INFO: Query returned 0 results
>>
>> There must be something special about your setup...
>>
>> -brian
>>
>> On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
>>
>>>
>>> Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
>>>
>>> i ran the following:
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
>>>
>>> I got the following file manager query:
>>> q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
>>>
>>> so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
>>>
>>> also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
>>>
>>> -brian
>>>
>>> On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>
>>>> Hey Brian,
>>>>
>>>> Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
>>>>
>>>> The exception I'm getting is this:
>>>>
>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
>>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>> Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
>>>> ... 2 more
>>>>
>>>> Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
>>>>
>>>> In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
>>>>
>>>> Thanks for your help!
>>>>
>>>> -Mike
>>>>
>>>> From: Brian Foster <ho...@mac.com>
>>>> Reply-To: <us...@oodt.apache.org>
>>>> Date: Fri, 18 May 2012 00:51:07 +0000
>>>> To: <us...@oodt.apache.org>
>>>> Subject: Re: Query Tool Bugs?
>>>>
>>>>
>>>> hey mike,
>>>>
>>>> You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
>>>>
>>>> RangeBeginningDate > '2007-01-01'
>>>>
>>>> Should be a space before and after >
>>>>
>>>> each value should start and end with single quote.
>>>>
>>>> -brian
>>>>
>>>> On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>
>>>>> Hey Brian,
>>>>>
>>>>> I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
>>>>>
>>>>> In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
>>>>>
>>>>> Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
>>>>>
>>>>> The SQLParser class liked the syntax and it produced the following post fix statement:
>>>>>
>>>>> Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
>>>>>
>>>>> I think from here, it creates some Boolean Query Criteria with the following content:
>>>>>
>>>>> Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
>>>>>
>>>>> I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
>>>>>
>>>>> I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
>>>>>
>>>>> SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
>>>>>
>>>>>
>>>>> -Mike
>>>>>
>>>>> From: Brian Foster <ho...@mac.com>
>>>>> Reply-To: <us...@oodt.apache.org>
>>>>> Date: Thu, 17 May 2012 18:40:20 +0000
>>>>> To: <us...@oodt.apache.org>
>>>>> Subject: Re: Query Tool Bugs?
>>>>>
>>>>>
>>>>> hey mike,
>>>>>
>>>>> would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
>>>>>
>>>>> -brian
>>>>>
>>>>> On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>
>>>>>> Hey Brian,
>>>>>>
>>>>>> Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
>>>>>>
>>>>>> In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
>>>>>>
>>>>>> Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
>>>>>>
>>>>>> ./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
>>>>>> May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
>>>>>> INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
>>>>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
>>>>>> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
>>>>>> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>>>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
>>>>>> at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
>>>>>> at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
>>>>>> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
>>>>>> ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
>>>>>> ', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>
>>>>>> Let me know if you'd like for me to file a bug for this.
>>>>>>
>>>>>> Thanks,
>>>>>> Mike
>>>>>>
>>>>>> From: Brian Foster <ho...@mac.com>
>>>>>> Reply-To: <us...@oodt.apache.org>
>>>>>> Date: Mon, 14 May 2012 22:28:58 +0000
>>>>>> To: <us...@oodt.apache.org>
>>>>>> Subject: Re: Query Tool Bugs?
>>>>>>
>>>>>> hey mike,
>>>>>>
>>>>>> first pass over the policy files look good... however, is the way you posted the query the way you are running it?
>>>>>>
>>>>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
>>>>>>
>>>>>> you have an extra set of double quotes around the where clause... try removing those
>>>>>>
>>>>>> -brian
>>>>>>
>>>>>> On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
>>>>>>
>>>>>>> Hey Chris,
>>>>>>>
>>>>>>> On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
>>>>>>> <ch...@jpl.nasa.gov> wrote:
>>>>>>>
>>>>>>> >Hey Mike,
>>>>>>> >
>>>>>>> >On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>>>>> >
>>>>>>> >>>>
>>>>>>> >>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>>>> >>>>Failed
>>>>>>> >>>> to perform complex query : You have an error in your SQL syntax; check
>>>>>>> >>>> the manual that corresponds to your MySQL server version for the right
>>>>>>> >>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>>>>> >>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>>>> >>>>(X
>>>>>>> >>>> mlRpcFileManagerClient.java:958)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:251)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>
>>>>>>> >>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>>>>>> >>> like a partial one, missing the timezone do you think that might
>>>>>>> >>> affect ir?
>>>>>>> >>
>>>>>>> >> I talked with Rishi regarding this and he recommended that the date and
>>>>>>> >> time be split when performing a query. Reason being is that the query
>>>>>>> >>tool
>>>>>>> >> blows up when trying to compare datetime values. He mentioned that he
>>>>>>> >> tried querying against ISO 8601 date/time values before and it didn't
>>>>>>> >>work
>>>>>>> >> for him and the way around it was to split it up. I think behind the
>>>>>>> >> scenes, the query tool is actually doing an ascii comparison, which
>>>>>>> >>might
>>>>>>> >> be why the tool might be having performance issues?
>>>>>>> >
>>>>>>> >Gotcha, that might help, yes. I was thinking: what is your repository
>>>>>>> >manager,
>>>>>>> >and catalog combination? If you are using e.g., a DataSourceCatalog,
>>>>>>> >with the XMLRepositoryManager, you'll need to turn on the quoteFields
>>>>>>> >option in the filemgr.properties for the DataSourceCatalog. This is
>>>>>>> >because,
>>>>>>> >in these scenarios, the identifier for elementIds is a string, compared
>>>>>>> >to a
>>>>>>> >number (which would be the case if you used the
>>>>>>> >DataSourceRepositoryManager --
>>>>>>> >the short answer there is don't, it's not as well maintained as the XML
>>>>>>> >one).
>>>>>>>
>>>>>>> My repository manager and catalog combination is defined as follows in the
>>>>>>> filemgr.properties:
>>>>>>>
>>>>>>> filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
>>>>>>> toryManagerFactory
>>>>>>>
>>>>>>> filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
>>>>>>> ogFactory
>>>>>>>
>>>>>>> org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> I tried surrounding the entire condition with quotes, but still no
>>>>>>> >>>>luck:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>>>>> >>>> RangeBeginningTime>'12:00:00.000Z'""
>>>>>>> >>>> Ambiguous output redirect.
>>>>>>> >>>>
>>>>>>> >>>> I'm assuming this is a syntax thing, although I don't know what the
>>>>>>> >>>> tool is expecting.
>>>>>>> >>>
>>>>>>> >>> Did you check the code in SVN?
>>>>>>> >>
>>>>>>> >> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>>>>>>> >> checked out onto my local machine. I can certainly build the trunk and
>>>>>>> >>see
>>>>>>> >> if I get the same results.
>>>>>>> >
>>>>>>> >I think there is a fix for something similar to this in the trunk (as
>>>>>>> >bfoster mentioned),
>>>>>>> >but thinking about this more, I bet you're having the quoteFields
>>>>>>> >problem, per
>>>>>>> >above. Scope it out and let me know.
>>>>>>>
>>>>>>> Based on the properties specified above, the quoteFields option is turned
>>>>>>> on. So I think the properties are set correctly, no?
>>>>>>> It'll be interesting to see if Brian was able to find an error in the
>>>>>>> policy files that I sent in the previous e-mail.
>>>>>>>
>>>>>>> >
>>>>>>> >>
>>>>>>> >>>
>>>>>>> >>>>
>>>>>>> >>>> My second issue that I'm running into is in regards to querying of
>>>>>>> >>>> dates. I tried the following query below and got the following output:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>>>>> >>>> String index out of range: -1
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>>>> >>>>1)
>>>>>>> >>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>>>>> >>>> at
>>>>>>> >>>>
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>>>> >>>>av
>>>>>>> >>>> a:255)
>>>>>>> >>>> at
>>>>>>> >>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>>>> >>>>
>>>>>>> >>>> For this particular product, I have 1 product in my catalog where the
>>>>>>> >>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>>>>> >>>> into why an exception is being thrown here. When I use an earlier date
>>>>>>> >>>> on my query, the tool returns a result as expected:
>>>>>>> >>>>
>>>>>>> >>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>>>>> >>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>>>>> >>>> log4j:WARN No appenders could be found for logger
>>>>>>> >>>> (org.apache.commons.httpclient.HttpClient).
>>>>>>> >>>> log4j:WARN Please initialize the log4j system properly.
>>>>>>> >>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>>>>> >>>> for more info.
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>>>> >>>>51
>>>>>>> >>>>
>>>>>>> >>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>>>> >>>>5-
>>>>>>> >>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>>>>> >>>
>>>>>>> >>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>>>>> >>>issue?
>>>>>>> >>>
>>>>>>> >>> Feel free to file a bug and would love you to investigate!
>>>>>>> >>
>>>>>>> >> I haven't dived into the code, but will certainly do this as SMAP will
>>>>>>> >> need these capabilities. I will file a bug if it turns out that this is
>>>>>>> >> indeed a bug.
>>>>>>> >
>>>>>>> >Great Mike, thanks.
>>>>>>> >
>>>>>>> >Cheers,
>>>>>>> >Chris
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mike
>>>>>>>
>>>>>>> >
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Chris Mattmann, Ph.D.
>>>>>>> >Senior Computer Scientist
>>>>>>> >NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>>>>>> >Office: 171-266B, Mailstop: 171-246
>>>>>>> >Email: chris.a.mattmann@nasa.gov
>>>>>>> >WWW: http://sunset.usc.edu/~mattmann/
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >Adjunct Assistant Professor, Computer Science Department
>>>>>>> >University of Southern California, Los Angeles, CA 90089 USA
>>>>>>> >++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> >
>>>>>>>
>>>> #!/bin/sh
>>>>
>>>> $JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
> <createProductTypeSQL.sh>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
ya looks like you got everything setup perfect... looks like you found a bug in DataSourceCatalog... i closed the bug you created and re-opened it as an issue to re-add support for MySql to the DataSourceCatalog
-brian
On May 22, 2012, at 08:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 22:32:30 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
ya looks like you got everything setup perfect... looks like you found a bug in DataSourceCatalog... i closed the bug you created and re-opened it as an issue to re-add support for MySql to the DataSourceCatalog
-brian
On May 22, 2012, at 08:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 22:32:30 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.�
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 22:32:30 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ahh, good catch! I tried this query with my mysql install and it produced the following message:
16:23:33 [(SELECT - 0 row(s), 0.000 secs] [Error Code: 1064, SQL State: 42000] You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
... 1 statement(s) executed, 0 row(s) affected, exec/fetch time: 0.000/0.000 sec [0 successful, 0 warnings, 1 errors]
Looking more into this issue, it appears that mysql doesn't support INTERSECT.
Eventually, the project I'm working on will be using Oracle instead of mysql, so this won't be an issue then.
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 22:54:48 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com>> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
i could be wrong, but a quick google search looks like INTERSECT is not supported in MySQL :( ... have to use INNER JOIN for it or something of the like... could be the problem you are seeing
-brian
On May 21, 2012, at 03:54 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
�
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.�
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
i could be wrong, but a quick google search looks like INTERSECT is not supported in MySQL :( ... have to use INNER JOIN for it or something of the like... could be the problem you are seeing
-brian
On May 21, 2012, at 03:54 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ahh, good catch! I tried this query with my mysql install and it produced the following message:
16:23:33 [(SELECT - 0 row(s), 0.000 secs] [Error Code: 1064, SQL State: 42000] You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
... 1 statement(s) executed, 0 row(s) affected, exec/fetch time: 0.000/0.000 sec [0 successful, 0 warnings, 1 errors]
Looking more into this issue, it appears that mysql doesn't support INTERSECT.
Eventually, the project I'm working on will be using Oracle instead of mysql, so this won't be an issue then.
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 22:54:48 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com>> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
�
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.�
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
try executing this directly in your mysql install and see what happens:
(SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
-brian
On May 21, 2012, at 03:32 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
element_id is set to a varchar. I'm using the attached shell script to create the database tables. Since I'm using XMLValidationLayer, does this mean that I should be creating a different set of tables? If so, what would those be? I'm assuming the provided cas-filemgr-schema-mysql.sql is used only if you're using DataSourceValidationLayer...
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 22:32:30 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.�
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
i copied in your etc files and policy files into a newly deployed trunk filemgr deployment... i then just commented out the code which talked to sql driver so i could see the SQL which would be sent to the sql driver... basically by doing that i've narrowed it down to something with your SQL connection... did you change element_id to a varchar instead of and int since you are using XMLValidationLayer... the provide cas-filemgr-schema-mysql.sql is setup to work with DataSourceValidationLayer... i think you are getting an error because there are quotes around your element_id and the database is expecting a integer (the quotes are correct in this case though, you DB is just configured wrong)
-brian
On May 21, 2012, at 02:42 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
mike can you post your SQL table CREATE statements
-brian
On May 21, 2012, at 12:12 PM, Brian Foster <ho...@mac.com> wrote:
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Hmmm...interesting.
I believe I attached a filemgr.properties in an earlier e-mail. Did you use this in your configuration as well?
Also, which version of the filemgr are you using on your computer? The trunk? Or the wengine-branch that you pointed me to earlier?
I'm also using a mysql DB with the file manager. I don't think that should matter though in this case, but I could be wrong. I'm assuming you set up your file manager with some other db backend?
Would it be possible to send me a tar package of your file manger deployment that you have on your computer? I want to see if I can just untar it on my computer, start it up and see if that works.
Thanks again for all your help on this,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 21 May 2012 19:12:09 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com>> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC�
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike... i'm kinda stumped... used your configuration on a filemgr on my computer and i get the following output:
$ ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.catalog.DataSourceCatalog paginateQuery
INFO: catalog query: executing: (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningTime' AND metadata_value > '12:00:00.000') INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id = 'urn:smap:product:RangeBeginningDate' AND metadata_value > '2007-01-01') ORDER BY product_id DESC
May 21, 2012 12:09:58 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager complexQuery
INFO: Query returned 0 results
There must be something special about your setup...
-brian
On May 21, 2012, at 11:27 AM, Brian Foster <ho...@mac.com> wrote:
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought) or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
Hey mike... i should have some time today to take a look into this more than just throwing out guesses... i put a print statement in the QueryTool code to see what the SqlParser was doing with the query...
i ran the following:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
I got the following file manager query:
q=AND(RangeBeginningTime:[12:00:00.000 TO null], RangeBeginningDate:[2007-01-01 TO null])
so it appears that it doesn't have anything to do with shell parsing (like i initially thought)�or SqlParser... i will take a closer look at your configuration today and let you know what i find.
also something that has been know to cause problems in the past is shell aliases... maybe take a look at those if you have any... in the mean time i will try and get to the bottom of this.
-brian
On May 21, 2012, at 10:02 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.�
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
#!/bin/sh
$JAVA_HOME/bin/java -Djava.ext.dirs=../lib org.apache.oodt.cas.filemgr.tools.QueryTool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate > '2007-01-01' AND RangeBeginningTime > '12:00:00.000'"
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
also... if putting the spaces still fails... i'm ready to bet this is a shell issue... instead of doing: ./query_tool --url .... �put the arguments inside query_tool script so you can just run it like: ./query_tool
-brian
On May 17, 2012, at 05:51 PM, Brian Foster <ho...@mac.com> wrote:
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Yep, I tried the query with spaces before and after > and also surrounded each value with a single quote and it didn't work for me. I also put the query into the shell script as you've suggested and still no luck.
The exception I'm getting is this:
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:951)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:250)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Caused by: org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:945)
... 2 more
Attached is the query tool script, which contains the query I'm trying. It appears that this is a bug. Since query tool has been deprecated in the trunk in favor of the super new file manager client tool, then I'll go ahead and file a JIRA issue for this against the file manager.
In the meantime, I'll definitely look more into this and see if I can find out where the bug is at.
Thanks for your help!
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 18 May 2012 00:51:07 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
You putting a space between the operators? its hard to tell in emails, stuff gets normalized sometimes.
RangeBeginningDate > '2007-01-01'
Should be a space before and after >
each value should start and end with single quote.
-brian
On May 17, 2012, at 04:18 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement:�['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria:�AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
I checked out the wengine-branch filemgr and unfortunately, it still didn't work. I tried a query using the wengine branch's query_tool since this version of the filemgr-client didn't have the ability to query.
In any case, I took the liberty of diving into the Query Tool code to see if I can find out more as to what is going on. So this is what I found:
Surrounding single quotes around each of the conditions got me further into the code. So, when I entered the query as follows,
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>2007-01-01' AND 'RangeBeginningTime>12:00:00Z'
The SQLParser class liked the syntax and it produced the following post fix statement:
Post Fix Statement: ['RangeBeginningDate>2007-01-01', 'RangeBeginningTime>12:00:00Z', AND]
I think from here, it creates some Boolean Query Criteria with the following content:
Boolean Query Criteria: AND('RangeBeginningTime:[12:00:00Z' TO null], 'RangeBeginningDate:[2007-01-01' TO null])
I noticed that there are unmatched single quotes around the date and time values. Not sure how that came about. The tool still threw an exception. I'm guessing it's because of the unmatched single quotes.
I tried surrounding the date and time values with single quotes as well, but the SQLParser class didn't like that either:
SELECT * FROM L0a_Radar WHERE 'RangeBeginningDate>'2007-01-01'' AND 'RangeBeginningTime>'12:00:00Z''
-Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Thu, 17 May 2012 18:40:20 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cheng, Cecilia S (388K)" <ce...@jpl.nasa.gov>.
Hi Mike / Brian,
I tried running a similar query in the wengine-branch/filemgr 's query_tool. We're using this catalog factory:
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.ColumnBasedDataSourceCatalogFactory
% ./query_tool --url $FILEMGR_URL --sql -query "SELECT PRODUCTID FROM OCO2_ECMWF WHERE AcquisitionDate>'2007-01-01' AND EndDateTime>'2010-09-24T00:00:00.000Z'"
This works. We don't have just a time field so I can't test with a query like Mike's.
Mike — are you sure the syntax that you used (RangeBeginningTime>'12:00:00Z') is valid? Maybe you want to break up your query to see whether it's failing on the RangeBeginningDate or RangeBeginningTime?
-- cecilia
From: Brian Foster <ho...@mac.com>>
Reply-To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Date: Thursday, May 17, 2012 11:40 AM
To: "user@oodt.apache.org<ma...@oodt.apache.org>" <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
would you mind trying to run this query using the wengine-branch filemgr (https://svn.apache.org/repos/asf/oodt/branches/wengine-branch/filemgr/)... there are a lot of bug fixes floating around in both Lucene and DataSource Catalogs in that branch... if it works with that then we know it is a bug with trunk.
-brian
On May 16, 2012, at 07:26 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>
Reply-To: <us...@oodt.apache.org>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Ya, I tried the query with and without the double quotes surrounding the where clause and they both didn't work for me.
In any case, I've built the latest file manager in the trunk and it fixed that StringIndexOutofBoundsException issue I was seeing. Thanks!
Out of curiosity, I tried using the latest File Manager Client tool with the query with multiple conditions to see if I could get a result, but it threw an exception as well:
./filemgr-client -u http://localhost:9000 -q "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00Z'" -op -sqlQuery
May 16, 2012 7:20:53 AM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:990)
at org.apache.oodt.cas.filemgr.cli.action.AbstractQueryCliAction.execute(AbstractQueryCliAction.java:75)
at org.apache.oodt.cas.cli.CmdLineUtility.execute(CmdLineUtility.java:331)
at org.apache.oodt.cas.cli.CmdLineUtility.run(CmdLineUtility.java:187)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.main(XmlRpcFileManagerClient.java:1323)
ERROR: Failed to perform sql query : sortBy 'null', outputFormat 'null', and delimiter '
', filterAlgor 'null', startDateTimeMetKey 'null', endDateTimeMetKey 'null', priorityMetKey 'null', null' : java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
Let me know if you'd like for me to file a bug for this.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Mon, 14 May 2012 22:28:58 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000<http://localhost:9000/> --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov>> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov<ma...@nasa.gov>
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
hey mike,
first pass over the policy files look good... however, is the way you posted the query the way you are running it?
./query_tool --url�http://localhost:9000�--sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
you have an extra set of double quotes around the where clause... try removing those
-brian
On May 14, 2012, at 09:32 AM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Chris,
On 5/12/12 11:25 AM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hey Mike,
>
>On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>
>>>>
>>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>>Failed
>>>> to perform complex query : You have an error in your SQL syntax; check
>>>> the manual that corresponds to your MySQL server version for the right
>>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery
>>>>(X
>>>> mlRpcFileManagerClient.java:958)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:251)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>> like a partial one, missing the timezone do you think that might
>>> affect ir?
>>
>> I talked with Rishi regarding this and he recommended that the date and
>> time be split when performing a query. Reason being is that the query
>>tool
>> blows up when trying to compare datetime values. He mentioned that he
>> tried querying against ISO 8601 date/time values before and it didn't
>>work
>> for him and the way around it was to split it up. I think behind the
>> scenes, the query tool is actually doing an ascii comparison, which
>>might
>> be why the tool might be having performance issues?
>
>Gotcha, that might help, yes. I was thinking: what is your repository
>manager,
>and catalog combination? If you are using e.g., a DataSourceCatalog,
>with the XMLRepositoryManager, you'll need to turn on the quoteFields
>option in the filemgr.properties for the DataSourceCatalog. This is
>because,
>in these scenarios, the identifier for elementIds is a string, compared
>to a
>number (which would be the case if you used the
>DataSourceRepositoryManager --
>the short answer there is don't, it's not as well maintained as the XML
>one).
My repository manager and catalog combination is defined as follows in the
filemgr.properties:
filemgr.repository.factory=org.apache.oodt.cas.filemgr.repository.XMLReposi
toryManagerFactory
filemgr.catalog.factory=org.apache.oodt.cas.filemgr.catalog.DataSourceCatal
ogFactory
org.apache.oodt.cas.filemgr.catalog.datasource.quoteFields=true
>
>>
>>>
>>>>
>>>> I tried surrounding the entire condition with quotes, but still no
>>>>luck:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>> RangeBeginningTime>'12:00:00.000Z'""
>>>> Ambiguous output redirect.
>>>>
>>>> I'm assuming this is a syntax thing, although I don't know what the
>>>> tool is expecting.
>>>
>>> Did you check the code in SVN?
>>
>> I'm running 0.3 of the code. Does the trunk fix this? I have the code
>> checked out onto my local machine. I can certainly build the trunk and
>>see
>> if I get the same results.
>
>I think there is a fix for something similar to this in the trunk (as
>bfoster mentioned),
>but thinking about this more, I bet you're having the quoteFields
>problem, per
>above. Scope it out and let me know.
Based on the properties specified above, the quoteFields option is turned
on. So I think the properties are set correctly, no?
It'll be interesting to see if Brian was able to find an error in the
policy files that I sent in the previous e-mail.
>
>>
>>>
>>>>
>>>> My second issue that I'm running into is in regards to querying of
>>>> dates. I tried the following query below and got the following output:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>> String index out of range: -1
>>>> at
>>>>
>>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:88
>>>>1)
>>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>>> at
>>>>
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.j
>>>>av
>>>> a:255)
>>>> at
>>>>org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>>
>>>> For this particular product, I have 1 product in my catalog where the
>>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>> into why an exception is being thrown here. When I use an earlier date
>>>> on my query, the tool returns a result as expected:
>>>>
>>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>>> log4j:WARN No appenders could be found for logger
>>>> (org.apache.commons.httpclient.HttpClient).
>>>> log4j:WARN Please initialize the log4j system properly.
>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>> for more info.
>>>>
>>>>
>>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20
>>>>51
>>>>
>>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.38
>>>>5-
>>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>>
>>> Interesting! Did you scope the code to see if there's a RangeQuery
>>>issue?
>>>
>>> Feel free to file a bug and would love you to investigate!
>>
>> I haven't dived into the code, but will certainly do this as SMAP will
>> need these capabilities. I will file a bug if it turns out that this is
>> indeed a bug.
>
>Great Mike, thanks.
>
>Cheers,
>Chris
Thanks,
Mike
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Mike,
On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>> to perform complex query : You have an error in your SQL syntax; check
>>> the manual that corresponds to your MySQL server version for the right
>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>> mlRpcFileManagerClient.java:958)
>>> at
>>> org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>> a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>> like a partial one, missing the timezone do you think that might
>> affect ir?
>
> I talked with Rishi regarding this and he recommended that the date and
> time be split when performing a query. Reason being is that the query tool
> blows up when trying to compare datetime values. He mentioned that he
> tried querying against ISO 8601 date/time values before and it didn't work
> for him and the way around it was to split it up. I think behind the
> scenes, the query tool is actually doing an ascii comparison, which might
> be why the tool might be having performance issues?
Gotcha, that might help, yes. I was thinking: what is your repository manager,
and catalog combination? If you are using e.g., a DataSourceCatalog,
with the XMLRepositoryManager, you'll need to turn on the quoteFields
option in the filemgr.properties for the DataSourceCatalog. This is because,
in these scenarios, the identifier for elementIds is a string, compared to a
number (which would be the case if you used the DataSourceRepositoryManager --
the short answer there is don't, it's not as well maintained as the XML one).
>
>>
>>>
>>> I tried surrounding the entire condition with quotes, but still no luck:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>> RangeBeginningTime>'12:00:00.000Z'""
>>> Ambiguous output redirect.
>>>
>>> I'm assuming this is a syntax thing, although I don't know what the
>>> tool is expecting.
>>
>> Did you check the code in SVN?
>
> I'm running 0.3 of the code. Does the trunk fix this? I have the code
> checked out onto my local machine. I can certainly build the trunk and see
> if I get the same results.
I think there is a fix for something similar to this in the trunk (as bfoster mentioned),
but thinking about this more, I bet you're having the quoteFields problem, per
above. Scope it out and let me know.
>
>>
>>>
>>> My second issue that I'm running into is in regards to querying of
>>> dates. I tried the following query below and got the following output:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> log4j:WARN No appenders could be found for logger
>>> (org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for more info.
>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>> String index out of range: -1
>>> at
>>> java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> at
>>> org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>> a:255)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> For this particular product, I have 1 product in my catalog where the
>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>> into why an exception is being thrown here. When I use an earlier date
>>> on my query, the tool returns a result as expected:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> log4j:WARN No appenders could be found for logger
>>> (org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for more info.
>>>
>>> /Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V2051
>>> 7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-
>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>
>> Interesting! Did you scope the code to see if there's a RangeQuery issue?
>>
>> Feel free to file a bug and would love you to investigate!
>
> I haven't dived into the code, but will certainly do this as SMAP will
> need these capabilities. I will file a bug if it turns out that this is
> indeed a bug.
Great Mike, thanks.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Mike,
On May 11, 2012, at 6:07 AM, Cayanan, Michael D (388J) wrote:
>>>
>>> org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>> to perform complex query : You have an error in your SQL syntax; check
>>> the manual that corresponds to your MySQL server version for the right
>>> syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>> L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>> org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>> mlRpcFileManagerClient.java:958)
>>> at
>>> org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>> a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>> Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>> like a partial one, missing the timezone do you think that might
>> affect ir?
>
> I talked with Rishi regarding this and he recommended that the date and
> time be split when performing a query. Reason being is that the query tool
> blows up when trying to compare datetime values. He mentioned that he
> tried querying against ISO 8601 date/time values before and it didn't work
> for him and the way around it was to split it up. I think behind the
> scenes, the query tool is actually doing an ascii comparison, which might
> be why the tool might be having performance issues?
Gotcha, that might help, yes. I was thinking: what is your repository manager,
and catalog combination? If you are using e.g., a DataSourceCatalog,
with the XMLRepositoryManager, you'll need to turn on the quoteFields
option in the filemgr.properties for the DataSourceCatalog. This is because,
in these scenarios, the identifier for elementIds is a string, compared to a
number (which would be the case if you used the DataSourceRepositoryManager --
the short answer there is don't, it's not as well maintained as the XML one).
>
>>
>>>
>>> I tried surrounding the entire condition with quotes, but still no luck:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>> RangeBeginningTime>'12:00:00.000Z'""
>>> Ambiguous output redirect.
>>>
>>> I'm assuming this is a syntax thing, although I don't know what the
>>> tool is expecting.
>>
>> Did you check the code in SVN?
>
> I'm running 0.3 of the code. Does the trunk fix this? I have the code
> checked out onto my local machine. I can certainly build the trunk and see
> if I get the same results.
I think there is a fix for something similar to this in the trunk (as bfoster mentioned),
but thinking about this more, I bet you're having the quoteFields problem, per
above. Scope it out and let me know.
>
>>
>>>
>>> My second issue that I'm running into is in regards to querying of
>>> dates. I tried the following query below and got the following output:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> log4j:WARN No appenders could be found for logger
>>> (org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for more info.
>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>> String index out of range: -1
>>> at
>>> java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> at
>>> org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>> a:255)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> For this particular product, I have 1 product in my catalog where the
>>> RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>> into why an exception is being thrown here. When I use an earlier date
>>> on my query, the tool returns a result as expected:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>> L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> log4j:WARN No appenders could be found for logger
>>> (org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>> for more info.
>>>
>>> /Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V2051
>>> 7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-
>>> 07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>
>> Interesting! Did you scope the code to see if there's a RangeQuery issue?
>>
>> Feel free to file a bug and would love you to investigate!
>
> I haven't dived into the code, but will certainly do this as SMAP will
> need these capabilities. I will file a bug if it turns out that this is
> indeed a bug.
Great Mike, thanks.
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by Cameron Goodale <go...@apache.org>.
Mike,
As my Lucene Catalog has started to explode in volume (catalog contains
over 290,000 docs) I have also needed to query the catalog in a more
complicated manner. (I am going to assume you are using the Lucene
Catalog, if not then read no further).
If you use the Luke Tool from Lucene[1] you can query the index in almost
any manner you can imagine. This tool is also extremely fast since it was
designed to connected directly to the index. I used the lukeall-1.0.1.jar
and it worked great. Hope this helps.
Cameron
[1] - http://bit.ly/JjRzu5
On Fri, May 11, 2012 at 11:29 AM, Verma, Rishi (388J) <
Rishi.Verma@jpl.nasa.gov> wrote:
> Hey Mike,
>
> Wanted to add some thoughts:
>
> >I talked with Rishi regarding this and he recommended that the date and
> >time be split when performing a query. Reason being is that the query tool
> >blows up when trying to compare datetime values. He mentioned that he
> >tried querying against ISO 8601 date/time values before and it didn't work
> >for him and the way around it was to split it up. I think behind the
> >scenes, the query tool is actually doing an ascii comparison, which might
> >be why the tool might be having performance issues?
>
>
> I was using the XML-RPC FileManager API directly to issue my queries (see
> [1] for RangeQuery API). The problem wasn't so much syntax errors, but
> performance issues. In other words, queries would hang and then eventually
> fail due to the large number of comparison checks being performed.
>
> +1 on the idea to split up your date for better search performance, but
> from your stacktrace, it looks like you have a syntax issue. Have you been
> able to test comparison (ie. use of '<' '>' etc) queries for non-time
> related metadata elements? That might be a good place to start, to see if
> you've got your syntax right.
>
> Thanks!
> Rishi
>
> --
> [1]
> http://svn.apache.org/repos/asf/oodt/tags/0.3/filemgr/src/main/java/org/apa
> che/oodt/cas/filemgr/structs/RangeQueryCriteria.java
>
> On 5/11/12 9:07 AM, "Cayanan, Michael D (388J)"
> <mi...@jpl.nasa.gov> wrote:
>
> >Hi Chris,
> >
> >Comments are below....
> >
> >On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
> ><ch...@jpl.nasa.gov> wrote:
> >
> >>Hi Mike,
> >>
> >>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
> >>
> >>> Hi All,
> >>>
> >>> I'm having several issues with the Query Tool and wondering if anyone
> >>>has run into these issues before:
> >>>
> >>> First, I'm having an issue when giving the Query Tool a query
> >>>containing multiple conditions:
> >>>
> >>> Below is a command-line run of my query:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
> >>>RangeBeginningTime>'12:00:00.000Z'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
> >>>to perform complex query : You have an error in your SQL syntax; check
> >>>the manual that corresponds to your MySQL server version for the right
> >>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
> >>>L0a_Radar_metadata WHERE element_id =' at line 1
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
> >>>i
> >>>entResponseProcessor.java:104)
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
> >>>e
> >>>ntResponseProcessor.java:71)
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> >>> at
> >>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
> >>>X
> >>>mlRpcFileManagerClient.java:952)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:251)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>> Exception in thread "main"
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >>>java.lang.Exception:
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
> >>>to perform complex query : You have an error in your SQL syntax; check
> >>>the manual that corresponds to your MySQL server version for the right
> >>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
> >>>L0a_Radar_metadata WHERE element_id =' at line 1
> >>> at
> >>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
> >>>X
> >>>mlRpcFileManagerClient.java:958)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:251)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>
> >>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
> >>like a partial one, missing the timezone do you think that might
> >>affect ir?
> >
> >I talked with Rishi regarding this and he recommended that the date and
> >time be split when performing a query. Reason being is that the query tool
> >blows up when trying to compare datetime values. He mentioned that he
> >tried querying against ISO 8601 date/time values before and it didn't work
> >for him and the way around it was to split it up. I think behind the
> >scenes, the query tool is actually doing an ascii comparison, which might
> >be why the tool might be having performance issues?
> >
> >>
> >>>
> >>> I tried surrounding the entire condition with quotes, but still no
> >>>luck:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
> >>>RangeBeginningTime>'12:00:00.000Z'""
> >>> Ambiguous output redirect.
> >>>
> >>> I'm assuming this is a syntax thing, although I don't know what the
> >>>tool is expecting.
> >>
> >>Did you check the code in SVN?
> >
> >I'm running 0.3 of the code. Does the trunk fix this? I have the code
> >checked out onto my local machine. I can certainly build the trunk and see
> >if I get the same results.
> >
> >>
> >>>
> >>> My second issue that I'm running into is in regards to querying of
> >>>dates. I tried the following query below and got the following output:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> >>>String index out of range: -1
> >>> at
> >>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881
> >>>)
> >>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:255)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>>
> >>> For this particular product, I have 1 product in my catalog where the
> >>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
> >>>into why an exception is being thrown here. When I use an earlier date
> >>>on my query, the tool returns a result as expected:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>>
> >>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V205
> >>>1
> >>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385
> >>>-
> >>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
> >>
> >>Interesting! Did you scope the code to see if there's a RangeQuery issue?
> >>
> >>Feel free to file a bug and would love you to investigate!
> >
> >I haven't dived into the code, but will certainly do this as SMAP will
> >need these capabilities. I will file a bug if it turns out that this is
> >indeed a bug.
> >
> >Thanks,
> >Mike
> >
> >>
> >>Cheers,
> >>Chris
> >>
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Chris Mattmann, Ph.D.
> >>Senior Computer Scientist
> >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>Office: 171-266B, Mailstop: 171-246
> >>Email: chris.a.mattmann@nasa.gov
> >>WWW: http://sunset.usc.edu/~mattmann/
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Adjunct Assistant Professor, Computer Science Department
> >>University of Southern California, Los Angeles, CA 90089 USA
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >
>
>
Re: Query Tool Bugs?
Posted by Cameron Goodale <go...@apache.org>.
Mike,
As my Lucene Catalog has started to explode in volume (catalog contains
over 290,000 docs) I have also needed to query the catalog in a more
complicated manner. (I am going to assume you are using the Lucene
Catalog, if not then read no further).
If you use the Luke Tool from Lucene[1] you can query the index in almost
any manner you can imagine. This tool is also extremely fast since it was
designed to connected directly to the index. I used the lukeall-1.0.1.jar
and it worked great. Hope this helps.
Cameron
[1] - http://bit.ly/JjRzu5
On Fri, May 11, 2012 at 11:29 AM, Verma, Rishi (388J) <
Rishi.Verma@jpl.nasa.gov> wrote:
> Hey Mike,
>
> Wanted to add some thoughts:
>
> >I talked with Rishi regarding this and he recommended that the date and
> >time be split when performing a query. Reason being is that the query tool
> >blows up when trying to compare datetime values. He mentioned that he
> >tried querying against ISO 8601 date/time values before and it didn't work
> >for him and the way around it was to split it up. I think behind the
> >scenes, the query tool is actually doing an ascii comparison, which might
> >be why the tool might be having performance issues?
>
>
> I was using the XML-RPC FileManager API directly to issue my queries (see
> [1] for RangeQuery API). The problem wasn't so much syntax errors, but
> performance issues. In other words, queries would hang and then eventually
> fail due to the large number of comparison checks being performed.
>
> +1 on the idea to split up your date for better search performance, but
> from your stacktrace, it looks like you have a syntax issue. Have you been
> able to test comparison (ie. use of '<' '>' etc) queries for non-time
> related metadata elements? That might be a good place to start, to see if
> you've got your syntax right.
>
> Thanks!
> Rishi
>
> --
> [1]
> http://svn.apache.org/repos/asf/oodt/tags/0.3/filemgr/src/main/java/org/apa
> che/oodt/cas/filemgr/structs/RangeQueryCriteria.java
>
> On 5/11/12 9:07 AM, "Cayanan, Michael D (388J)"
> <mi...@jpl.nasa.gov> wrote:
>
> >Hi Chris,
> >
> >Comments are below....
> >
> >On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
> ><ch...@jpl.nasa.gov> wrote:
> >
> >>Hi Mike,
> >>
> >>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
> >>
> >>> Hi All,
> >>>
> >>> I'm having several issues with the Query Tool and wondering if anyone
> >>>has run into these issues before:
> >>>
> >>> First, I'm having an issue when giving the Query Tool a query
> >>>containing multiple conditions:
> >>>
> >>> Below is a command-line run of my query:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
> >>>RangeBeginningTime>'12:00:00.000Z'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
> >>>to perform complex query : You have an error in your SQL syntax; check
> >>>the manual that corresponds to your MySQL server version for the right
> >>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
> >>>L0a_Radar_metadata WHERE element_id =' at line 1
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
> >>>i
> >>>entResponseProcessor.java:104)
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
> >>>e
> >>>ntResponseProcessor.java:71)
> >>> at
> >>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> >>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> >>> at
> >>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
> >>>X
> >>>mlRpcFileManagerClient.java:952)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:251)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>> Exception in thread "main"
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
> >>>java.lang.Exception:
> >>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
> >>>to perform complex query : You have an error in your SQL syntax; check
> >>>the manual that corresponds to your MySQL server version for the right
> >>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
> >>>L0a_Radar_metadata WHERE element_id =' at line 1
> >>> at
> >>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
> >>>X
> >>>mlRpcFileManagerClient.java:958)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:251)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>
> >>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
> >>like a partial one, missing the timezone do you think that might
> >>affect ir?
> >
> >I talked with Rishi regarding this and he recommended that the date and
> >time be split when performing a query. Reason being is that the query tool
> >blows up when trying to compare datetime values. He mentioned that he
> >tried querying against ISO 8601 date/time values before and it didn't work
> >for him and the way around it was to split it up. I think behind the
> >scenes, the query tool is actually doing an ascii comparison, which might
> >be why the tool might be having performance issues?
> >
> >>
> >>>
> >>> I tried surrounding the entire condition with quotes, but still no
> >>>luck:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
> >>>RangeBeginningTime>'12:00:00.000Z'""
> >>> Ambiguous output redirect.
> >>>
> >>> I'm assuming this is a syntax thing, although I don't know what the
> >>>tool is expecting.
> >>
> >>Did you check the code in SVN?
> >
> >I'm running 0.3 of the code. Does the trunk fix this? I have the code
> >checked out onto my local machine. I can certainly build the trunk and see
> >if I get the same results.
> >
> >>
> >>>
> >>> My second issue that I'm running into is in regards to querying of
> >>>dates. I tried the following query below and got the following output:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
> >>>String index out of range: -1
> >>> at
> >>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881
> >>>)
> >>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
> >>> at
> >>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
> >>>v
> >>>a:255)
> >>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> >>>
> >>> For this particular product, I have 1 product in my catalog where the
> >>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
> >>>into why an exception is being thrown here. When I use an earlier date
> >>>on my query, the tool returns a result as expected:
> >>>
> >>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
> >>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
> >>> log4j:WARN No appenders could be found for logger
> >>>(org.apache.commons.httpclient.HttpClient).
> >>> log4j:WARN Please initialize the log4j system properly.
> >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
> >>>for more info.
> >>>
> >>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V205
> >>>1
> >>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385
> >>>-
> >>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
> >>
> >>Interesting! Did you scope the code to see if there's a RangeQuery issue?
> >>
> >>Feel free to file a bug and would love you to investigate!
> >
> >I haven't dived into the code, but will certainly do this as SMAP will
> >need these capabilities. I will file a bug if it turns out that this is
> >indeed a bug.
> >
> >Thanks,
> >Mike
> >
> >>
> >>Cheers,
> >>Chris
> >>
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Chris Mattmann, Ph.D.
> >>Senior Computer Scientist
> >>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> >>Office: 171-266B, Mailstop: 171-246
> >>Email: chris.a.mattmann@nasa.gov
> >>WWW: http://sunset.usc.edu/~mattmann/
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>Adjunct Assistant Professor, Computer Science Department
> >>University of Southern California, Los Angeles, CA 90089 USA
> >>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >>
> >
>
>
Re: Query Tool Bugs?
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hey Mike,
Wanted to add some thoughts:
>I talked with Rishi regarding this and he recommended that the date and
>time be split when performing a query. Reason being is that the query tool
>blows up when trying to compare datetime values. He mentioned that he
>tried querying against ISO 8601 date/time values before and it didn't work
>for him and the way around it was to split it up. I think behind the
>scenes, the query tool is actually doing an ascii comparison, which might
>be why the tool might be having performance issues?
I was using the XML-RPC FileManager API directly to issue my queries (see
[1] for RangeQuery API). The problem wasn't so much syntax errors, but
performance issues. In other words, queries would hang and then eventually
fail due to the large number of comparison checks being performed.
+1 on the idea to split up your date for better search performance, but
from your stacktrace, it looks like you have a syntax issue. Have you been
able to test comparison (ie. use of '<' '>' etc) queries for non-time
related metadata elements? That might be a good place to start, to see if
you've got your syntax right.
Thanks!
Rishi
--
[1]
http://svn.apache.org/repos/asf/oodt/tags/0.3/filemgr/src/main/java/org/apa
che/oodt/cas/filemgr/structs/RangeQueryCriteria.java
On 5/11/12 9:07 AM, "Cayanan, Michael D (388J)"
<mi...@jpl.nasa.gov> wrote:
>Hi Chris,
>
>Comments are below....
>
>On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
><ch...@jpl.nasa.gov> wrote:
>
>>Hi Mike,
>>
>>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi All,
>>>
>>> I'm having several issues with the Query Tool and wondering if anyone
>>>has run into these issues before:
>>>
>>> First, I'm having an issue when giving the Query Tool a query
>>>containing multiple conditions:
>>>
>>> Below is a command-line run of my query:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>i
>>>entResponseProcessor.java:104)
>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>e
>>>ntResponseProcessor.java:71)
>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>X
>>>mlRpcFileManagerClient.java:952)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> Exception in thread "main"
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>X
>>>mlRpcFileManagerClient.java:958)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>like a partial one, missing the timezone do you think that might
>>affect ir?
>
>I talked with Rishi regarding this and he recommended that the date and
>time be split when performing a query. Reason being is that the query tool
>blows up when trying to compare datetime values. He mentioned that he
>tried querying against ISO 8601 date/time values before and it didn't work
>for him and the way around it was to split it up. I think behind the
>scenes, the query tool is actually doing an ascii comparison, which might
>be why the tool might be having performance issues?
>
>>
>>>
>>> I tried surrounding the entire condition with quotes, but still no
>>>luck:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'""
>>> Ambiguous output redirect.
>>>
>>> I'm assuming this is a syntax thing, although I don't know what the
>>>tool is expecting.
>>
>>Did you check the code in SVN?
>
>I'm running 0.3 of the code. Does the trunk fix this? I have the code
>checked out onto my local machine. I can certainly build the trunk and see
>if I get the same results.
>
>>
>>>
>>> My second issue that I'm running into is in regards to querying of
>>>dates. I tried the following query below and got the following output:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>String index out of range: -1
>>> at
>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881
>>>)
>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:255)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> For this particular product, I have 1 product in my catalog where the
>>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>into why an exception is being thrown here. When I use an earlier date
>>>on my query, the tool returns a result as expected:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>>
>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V205
>>>1
>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385
>>>-
>>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>
>>Interesting! Did you scope the code to see if there's a RangeQuery issue?
>>
>>Feel free to file a bug and would love you to investigate!
>
>I haven't dived into the code, but will certainly do this as SMAP will
>need these capabilities. I will file a bug if it turns out that this is
>indeed a bug.
>
>Thanks,
>Mike
>
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW: http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
Re: Query Tool Bugs?
Posted by "Verma, Rishi (388J)" <Ri...@jpl.nasa.gov>.
Hey Mike,
Wanted to add some thoughts:
>I talked with Rishi regarding this and he recommended that the date and
>time be split when performing a query. Reason being is that the query tool
>blows up when trying to compare datetime values. He mentioned that he
>tried querying against ISO 8601 date/time values before and it didn't work
>for him and the way around it was to split it up. I think behind the
>scenes, the query tool is actually doing an ascii comparison, which might
>be why the tool might be having performance issues?
I was using the XML-RPC FileManager API directly to issue my queries (see
[1] for RangeQuery API). The problem wasn't so much syntax errors, but
performance issues. In other words, queries would hang and then eventually
fail due to the large number of comparison checks being performed.
+1 on the idea to split up your date for better search performance, but
from your stacktrace, it looks like you have a syntax issue. Have you been
able to test comparison (ie. use of '<' '>' etc) queries for non-time
related metadata elements? That might be a good place to start, to see if
you've got your syntax right.
Thanks!
Rishi
--
[1]
http://svn.apache.org/repos/asf/oodt/tags/0.3/filemgr/src/main/java/org/apa
che/oodt/cas/filemgr/structs/RangeQueryCriteria.java
On 5/11/12 9:07 AM, "Cayanan, Michael D (388J)"
<mi...@jpl.nasa.gov> wrote:
>Hi Chris,
>
>Comments are below....
>
>On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
><ch...@jpl.nasa.gov> wrote:
>
>>Hi Mike,
>>
>>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
>>
>>> Hi All,
>>>
>>> I'm having several issues with the Query Tool and wondering if anyone
>>>has run into these issues before:
>>>
>>> First, I'm having an issue when giving the Query Tool a query
>>>containing multiple conditions:
>>>
>>> Below is a command-line run of my query:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCl
>>>i
>>>entResponseProcessor.java:104)
>>> at
>>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcCli
>>>e
>>>ntResponseProcessor.java:71)
>>> at
>>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>X
>>>mlRpcFileManagerClient.java:952)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>> Exception in thread "main"
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>>java.lang.Exception:
>>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>>to perform complex query : You have an error in your SQL syntax; check
>>>the manual that corresponds to your MySQL server version for the right
>>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>>L0a_Radar_metadata WHERE element_id =' at line 1
>>> at
>>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(
>>>X
>>>mlRpcFileManagerClient.java:958)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:251)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>>like a partial one, missing the timezone do you think that might
>>affect ir?
>
>I talked with Rishi regarding this and he recommended that the date and
>time be split when performing a query. Reason being is that the query tool
>blows up when trying to compare datetime values. He mentioned that he
>tried querying against ISO 8601 date/time values before and it didn't work
>for him and the way around it was to split it up. I think behind the
>scenes, the query tool is actually doing an ascii comparison, which might
>be why the tool might be having performance issues?
>
>>
>>>
>>> I tried surrounding the entire condition with quotes, but still no
>>>luck:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>>RangeBeginningTime>'12:00:00.000Z'""
>>> Ambiguous output redirect.
>>>
>>> I'm assuming this is a syntax thing, although I don't know what the
>>>tool is expecting.
>>
>>Did you check the code in SVN?
>
>I'm running 0.3 of the code. Does the trunk fix this? I have the code
>checked out onto my local machine. I can certainly build the trunk and see
>if I get the same results.
>
>>
>>>
>>> My second issue that I'm running into is in regards to querying of
>>>dates. I tried the following query below and got the following output:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>>String index out of range: -1
>>> at
>>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881
>>>)
>>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>>> at
>>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.ja
>>>v
>>>a:255)
>>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>>
>>> For this particular product, I have 1 product in my catalog where the
>>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>>into why an exception is being thrown here. When I use an earlier date
>>>on my query, the tool returns a result as expected:
>>>
>>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>>> log4j:WARN No appenders could be found for logger
>>>(org.apache.commons.httpclient.HttpClient).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>>for more info.
>>>
>>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V205
>>>1
>>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385
>>>-
>>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>>
>>Interesting! Did you scope the code to see if there's a RangeQuery issue?
>>
>>Feel free to file a bug and would love you to investigate!
>
>I haven't dived into the code, but will certainly do this as SMAP will
>need these capabilities. I will file a bug if it turns out that this is
>indeed a bug.
>
>Thanks,
>Mike
>
>>
>>Cheers,
>>Chris
>>
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Chris Mattmann, Ph.D.
>>Senior Computer Scientist
>>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>>Office: 171-266B, Mailstop: 171-246
>>Email: chris.a.mattmann@nasa.gov
>>WWW: http://sunset.usc.edu/~mattmann/
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>Adjunct Assistant Professor, Computer Science Department
>>University of Southern California, Los Angeles, CA 90089 USA
>>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Chris,
Comments are below....
On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi All,
>>
>> I'm having several issues with the Query Tool and wondering if anyone
>>has run into these issues before:
>>
>> First, I'm having an issue when giving the Query Tool a query
>>containing multiple conditions:
>>
>> Below is a command-line run of my query:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>RangeBeginningTime>'12:00:00.000Z'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>to perform complex query : You have an error in your SQL syntax; check
>>the manual that corresponds to your MySQL server version for the right
>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>L0a_Radar_metadata WHERE element_id =' at line 1
>> at
>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCli
>>entResponseProcessor.java:104)
>> at
>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClie
>>ntResponseProcessor.java:71)
>> at
>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>mlRpcFileManagerClient.java:952)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:251)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> Exception in thread "main"
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>java.lang.Exception:
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>to perform complex query : You have an error in your SQL syntax; check
>>the manual that corresponds to your MySQL server version for the right
>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>L0a_Radar_metadata WHERE element_id =' at line 1
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>mlRpcFileManagerClient.java:958)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:251)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>
>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>like a partial one, missing the timezone do you think that might
>affect ir?
I talked with Rishi regarding this and he recommended that the date and
time be split when performing a query. Reason being is that the query tool
blows up when trying to compare datetime values. He mentioned that he
tried querying against ISO 8601 date/time values before and it didn't work
for him and the way around it was to split it up. I think behind the
scenes, the query tool is actually doing an ascii comparison, which might
be why the tool might be having performance issues?
>
>>
>> I tried surrounding the entire condition with quotes, but still no luck:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>RangeBeginningTime>'12:00:00.000Z'""
>> Ambiguous output redirect.
>>
>> I'm assuming this is a syntax thing, although I don't know what the
>>tool is expecting.
>
>Did you check the code in SVN?
I'm running 0.3 of the code. Does the trunk fix this? I have the code
checked out onto my local machine. I can certainly build the trunk and see
if I get the same results.
>
>>
>> My second issue that I'm running into is in regards to querying of
>>dates. I tried the following query below and got the following output:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>String index out of range: -1
>> at
>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:255)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>> For this particular product, I have 1 product in my catalog where the
>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>into why an exception is being thrown here. When I use an earlier date
>>on my query, the tool returns a result as expected:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>>
>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V2051
>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-
>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>
>Interesting! Did you scope the code to see if there's a RangeQuery issue?
>
>Feel free to file a bug and would love you to investigate!
I haven't dived into the code, but will certainly do this as SMAP will
need these capabilities. I will file a bug if it turns out that this is
indeed a bug.
Thanks,
Mike
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hi Chris,
Comments are below....
On 5/10/12 6:22 PM, "Mattmann, Chris A (388J)"
<ch...@jpl.nasa.gov> wrote:
>Hi Mike,
>
>On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
>
>> Hi All,
>>
>> I'm having several issues with the Query Tool and wondering if anyone
>>has run into these issues before:
>>
>> First, I'm having an issue when giving the Query Tool a query
>>containing multiple conditions:
>>
>> Below is a command-line run of my query:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND
>>RangeBeginningTime>'12:00:00.000Z'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>> org.apache.xmlrpc.XmlRpcException: java.lang.Exception:
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>to perform complex query : You have an error in your SQL syntax; check
>>the manual that corresponds to your MySQL server version for the right
>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>L0a_Radar_metadata WHERE element_id =' at line 1
>> at
>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcCli
>>entResponseProcessor.java:104)
>> at
>>org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClie
>>ntResponseProcessor.java:71)
>> at
>>org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
>> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>mlRpcFileManagerClient.java:952)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:251)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>> Exception in thread "main"
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException:
>>java.lang.Exception:
>>org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed
>>to perform complex query : You have an error in your SQL syntax; check
>>the manual that corresponds to your MySQL server version for the right
>>syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM
>>L0a_Radar_metadata WHERE element_id =' at line 1
>> at
>>org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(X
>>mlRpcFileManagerClient.java:958)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:251)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>
>Just out of curiosity, is that correct ISO 8601 date/time format? Looks
>like a partial one, missing the timezone do you think that might
>affect ir?
I talked with Rishi regarding this and he recommended that the date and
time be split when performing a query. Reason being is that the query tool
blows up when trying to compare datetime values. He mentioned that he
tried querying against ISO 8601 date/time values before and it didn't work
for him and the way around it was to split it up. I think behind the
scenes, the query tool is actually doing an ascii comparison, which might
be why the tool might be having performance issues?
>
>>
>> I tried surrounding the entire condition with quotes, but still no luck:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND
>>RangeBeginningTime>'12:00:00.000Z'""
>> Ambiguous output redirect.
>>
>> I'm assuming this is a syntax thing, although I don't know what the
>>tool is expecting.
>
>Did you check the code in SVN?
I'm running 0.3 of the code. Does the trunk fix this? I have the code
checked out onto my local machine. I can certainly build the trunk and see
if I get the same results.
>
>>
>> My second issue that I'm running into is in regards to querying of
>>dates. I tried the following query below and got the following output:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>> Exception in thread "main" java.lang.StringIndexOutOfBoundsException:
>>String index out of range: -1
>> at
>>java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
>> at java.lang.StringBuffer.substring(StringBuffer.java:416)
>> at
>>org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.jav
>>a:255)
>> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>>
>> For this particular product, I have 1 product in my catalog where the
>>RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors
>>into why an exception is being thrown here. When I use an earlier date
>>on my query, the tool returns a result as expected:
>>
>> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM
>>L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
>> log4j:WARN No appenders could be found for logger
>>(org.apache.commons.httpclient.HttpClient).
>> log4j:WARN Please initialize the log4j system properly.
>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig
>>for more info.
>>
>>/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V2051
>>7SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-
>>07:00,L0a_Radar,23:30:25.000Z,2007-03-01
>
>Interesting! Did you scope the code to see if there's a RangeQuery issue?
>
>Feel free to file a bug and would love you to investigate!
I haven't dived into the code, but will certainly do this as SMAP will
need these capabilities. I will file a bug if it turns out that this is
indeed a bug.
Thanks,
Mike
>
>Cheers,
>Chris
>
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Chris Mattmann, Ph.D.
>Senior Computer Scientist
>NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>Office: 171-266B, Mailstop: 171-246
>Email: chris.a.mattmann@nasa.gov
>WWW: http://sunset.usc.edu/~mattmann/
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>Adjunct Assistant Professor, Computer Science Department
>University of Southern California, Los Angeles, CA 90089 USA
>++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
> Hi All,
>
> I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
>
> First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
>
> Below is a command-line run of my query:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Just out of curiosity, is that correct ISO 8601 date/time format? Looks like a partial one, missing the timezone do you think that might
affect ir?
>
> I tried surrounding the entire condition with quotes, but still no luck:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
> Ambiguous output redirect.
>
> I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
Did you check the code in SVN?
>
> My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
> at java.lang.StringBuffer.substring(StringBuffer.java:416)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>
> For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> /Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Interesting! Did you scope the code to see if there's a RangeQuery issue?
Feel free to file a bug and would love you to investigate!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Mike,
On May 10, 2012, at 1:28 PM, Cayanan, Michael D (388J) wrote:
> Hi All,
>
> I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
>
> First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
>
> Below is a command-line run of my query:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
> at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
> at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
> at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
> Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
> at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Just out of curiosity, is that correct ISO 8601 date/time format? Looks like a partial one, missing the timezone do you think that might
affect ir?
>
> I tried surrounding the entire condition with quotes, but still no luck:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
> Ambiguous output redirect.
>
> I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
Did you check the code in SVN?
>
> My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
> at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
> at java.lang.StringBuffer.substring(StringBuffer.java:416)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
> at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
>
> For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
>
> ./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
> log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> /Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Interesting! Did you scope the code to see if there's a RangeQuery issue?
Feel free to file a bug and would love you to investigate!
Cheers,
Chris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Attached is a zip of the policy. Apologies for the many files. :) There's a lot of inheritance going on, so I figured it'd be easier to just attach the entire policy for completeness sake.
Let me know if you see anything wrong with how RangeBeginningTime is specified. It appears to be correct from what I can see, but maybe I might be missing something here.
The L0a_Radar product type is defined under policy/smap-product/product-types.xml. policy/smap-product/product-type-element-map.xml shows what parent L0a_Radar maps to.
Regarding the 2nd problem, great to hear that this is resolved in the trunk. I will definitely check this out.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 11 May 2012 21:25:14 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
I think there are 2 different problems here... see comment below inline:
On May 10, 2012, at 01:28 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hi All,
I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
Below is a command-line run of my query:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
can you post your policy files... Looks like you may have a problem with RangeBeginningTime
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
I tried surrounding the entire condition with quotes, but still no luck:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
Ambiguous output redirect.
I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
at java.lang.StringBuffer.substring(StringBuffer.java:416)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
This is an issue which was resolve in the truck... has to do with no results being found... use SqlQuery CLI action.
For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Thanks,
Mike
Re: Query Tool Bugs?
Posted by "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>.
Hey Brian,
Attached is a zip of the policy. Apologies for the many files. :) There's a lot of inheritance going on, so I figured it'd be easier to just attach the entire policy for completeness sake.
Let me know if you see anything wrong with how RangeBeginningTime is specified. It appears to be correct from what I can see, but maybe I might be missing something here.
The L0a_Radar product type is defined under policy/smap-product/product-types.xml. policy/smap-product/product-type-element-map.xml shows what parent L0a_Radar maps to.
Regarding the 2nd problem, great to hear that this is resolved in the trunk. I will definitely check this out.
Thanks,
Mike
From: Brian Foster <ho...@mac.com>>
Reply-To: <us...@oodt.apache.org>>
Date: Fri, 11 May 2012 21:25:14 +0000
To: <us...@oodt.apache.org>>
Subject: Re: Query Tool Bugs?
I think there are 2 different problems here... see comment below inline:
On May 10, 2012, at 01:28 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov>> wrote:
Hi All,
I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
Below is a command-line run of my query:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
can you post your policy files... Looks like you may have a problem with RangeBeginningTime
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
I tried surrounding the entire condition with quotes, but still no luck:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
Ambiguous output redirect.
I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
at java.lang.StringBuffer.substring(StringBuffer.java:416)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
This is an issue which was resolve in the truck... has to do with no results being found... use SqlQuery CLI action.
For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Thanks,
Mike
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
I think there are 2 different problems here... see comment below inline:
On May 10, 2012, at 01:28 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi All,
I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
Below is a command-line run of my query:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
can you post your policy files... Looks like you may have a problem with RangeBeginningTime
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
I tried surrounding the entire condition with quotes, but still no luck:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
Ambiguous output redirect.
I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
at java.lang.StringBuffer.substring(StringBuffer.java:416)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
This is an issue which was resolve in the truck... has to do with no results being found... use SqlQuery CLI action.
For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here. When I use an earlier date on my query, the tool returns a result as expected:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Thanks,
Mike
Re: Query Tool Bugs?
Posted by Brian Foster <ho...@mac.com>.
I think there are 2 different problems here... see comment below inline:
On May 10, 2012, at 01:28 PM, "Cayanan, Michael D (388J)" <mi...@jpl.nasa.gov> wrote:
Hi All,
I'm having several issues with the Query Tool and wondering if anyone has run into these issues before:
First, I'm having an issue when giving the Query Tool a query containing multiple conditions:
Below is a command-line run of my query:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
org.apache.xmlrpc.XmlRpcException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
�
can you post your policy files... Looks like you may have a problem with RangeBeginningTime
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeException(XmlRpcClientResponseProcessor.java:104)
at org.apache.xmlrpc.XmlRpcClientResponseProcessor.decodeResponse(XmlRpcClientResponseProcessor.java:71)
at org.apache.xmlrpc.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:73)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:194)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:185)
at org.apache.xmlrpc.XmlRpcClient.execute(XmlRpcClient.java:178)
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:952)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
Exception in thread "main" org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: java.lang.Exception: org.apache.oodt.cas.filemgr.structs.exceptions.CatalogException: Failed to perform complex query : You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'INTERSECT (SELECT DISTINCT product_id FROM L0a_Radar_metadata WHERE element_id =' at line 1
at org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient.complexQuery(XmlRpcFileManagerClient.java:958)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:251)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
I tried surrounding the entire condition with quotes, but still no luck:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE "RangeBeginningDate>'2007-01-01' AND RangeBeginningTime>'12:00:00.000Z'""
Ambiguous output redirect.
I'm assuming this is a syntax thing, although I don't know what the tool is expecting.
My second issue that I'm running into is in regards to querying of dates. I tried the following query below and got the following output:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-03-02'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.StringIndexOutOfBoundsException: String index out of range: -1
at java.lang.AbstractStringBuilder.substring(AbstractStringBuilder.java:881)
at java.lang.StringBuffer.substring(StringBuffer.java:416)
at org.apache.oodt.cas.filemgr.tools.QueryTool.performSqlQuery(QueryTool.java:255)
at org.apache.oodt.cas.filemgr.tools.QueryTool.main(QueryTool.java:241)
�
This is an issue which was resolve in the truck... has to do with no results being found... use SqlQuery CLI action.
For this particular product, I have 1 product in my catalog where the RangeBeginningDate is equal to '2007-03-01'. Not sure if that factors into why an exception is being thrown here.�When I use an earlier date on my query, the tool returns a result as expected:
./query_tool --url http://localhost:9000 --sql -query "SELECT * FROM L0a_Radar WHERE RangeBeginningDate>'2007-01-01'"
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
/Users/mcayanan/smap/staging,2007-03-01,23:30:25.000Z,314,L0a_Radar,V20517SGS0706023302501.VCD,V20517SGS0706023302501.VCD,2012-05-08T14:27:59.385-07:00,L0a_Radar,23:30:25.000Z,2007-03-01
Thanks,
Mike