You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Abdul Chaudhry (JIRA)" <ji...@apache.org> on 2009/07/07 00:39:14 UTC
[jira] Created: (SOLR-1262) DIH needs support for prepared
statements
DIH needs support for prepared statements
------------------------------------------
Key: SOLR-1262
URL: https://issues.apache.org/jira/browse/SOLR-1262
Project: Solr
Issue Type: Improvement
Components: contrib - DataImportHandler
Affects Versions: 1.3
Environment: linux
mysql
Reporter: Abdul Chaudhry
Priority: Critical
During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
The queries we are using are complex and involve joins spanning across multiple tables.
We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-1262:
------------------------------
Priority: Minor (was: Critical)
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Abdul Chaudhry (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733138#action_12733138 ]
Abdul Chaudhry commented on SOLR-1262:
--------------------------------------
I could try to use stored procedures instead of trying to get prepared stements to work in the DIH - however that would mean most of the logic would need to live in SQL and I hate SQL.
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Abdul Chaudhry (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733144#action_12733144 ]
Abdul Chaudhry commented on SOLR-1262:
--------------------------------------
That's fine. Support for stored procedures would be great. I am not sure what code changes would be involved in the DIH - I guess you would need a CDATA section in the XML data-config file - we are currently using MYSQL 5.0.something
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1262) DIH needs support for callable
statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1262:
-----------------------------
Summary: DIH needs support for callable statements (was: DIH needs support for prepared statements )
> DIH needs support for callable statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733141#action_12733141 ]
Noble Paul commented on SOLR-1262:
----------------------------------
bq. however that would mean most of the logic would need to live in SQL and I hate SQL.
prepared statement is essentially an SQL statement with placeholders . I do not see any value add by supporting PreparedStatements
But stored procedure support is something we can look into
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732846#action_12732846 ]
Noble Paul commented on SOLR-1262:
----------------------------------
do you mean prepared statement or callablestatement (stored procedures)? . I guess there is not much difference between the semantics of prepared statements and sql statements
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1262) DIH needs support for callable
statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733148#action_12733148 ]
Noble Paul commented on SOLR-1262:
----------------------------------
The syntax of a callable statement is distinct from that of a select . So we can just use that aspect to figure out if it is a callable statement
we can support callable statements with a few restrictions
* Only IN args support
{code}
query="{call getResult( ${x.somevar1} ,${y.somevar2})}"
{code}
Before invoking the callable statements the placeholder locations will be replaced with '?'
> DIH needs support for callable statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-1262) DIH needs support for prepared
statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul reassigned SOLR-1262:
--------------------------------
Assignee: Noble Paul
> DIH needs support for prepared statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1262) DIH needs support for callable
statements
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-1262:
----------------------------------------
Fix Version/s: 1.5
> DIH needs support for callable statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
> Fix For: 1.5
>
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SOLR-1262) DIH needs support for
callable statements
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733148#action_12733148 ]
Noble Paul edited comment on SOLR-1262 at 7/20/09 12:35 AM:
------------------------------------------------------------
The syntax of a callable statement is distinct from that of a select . So we can just use that aspect to figure out if it is a callable statement
we can support callable statements with a few restrictions
* Only IN args support
* no return types
* probably no multiple resultsets too
{code}
query="{call getResult( ${x.somevar1} ,${y.somevar2})}"
{code}
Before invoking the callable statements the placeholder locations will be replaced with '?'
as
{code}
{call getResult( ?,?)}
{code}
and setxxx() can be called with the values of ${x.somevar1} and ${y.somevar2}
was (Author: noble.paul):
The syntax of a callable statement is distinct from that of a select . So we can just use that aspect to figure out if it is a callable statement
we can support callable statements with a few restrictions
* Only IN args support
{code}
query="{call getResult( ${x.somevar1} ,${y.somevar2})}"
{code}
Before invoking the callable statements the placeholder locations will be replaced with '?'
> DIH needs support for callable statements
> ------------------------------------------
>
> Key: SOLR-1262
> URL: https://issues.apache.org/jira/browse/SOLR-1262
> Project: Solr
> Issue Type: Improvement
> Components: contrib - DataImportHandler
> Affects Versions: 1.3
> Environment: linux
> mysql
> Reporter: Abdul Chaudhry
> Assignee: Noble Paul
> Priority: Minor
>
> During an indexing run we noticed that we were spending a lot of time creating and tearing down queries in mysql
> The queries we are using are complex and involve joins spanning across multiple tables.
> We should support prepared statements in the data import handler via the data-config.xml file - for those databases that support prepared statements.
> We could add a new attribute to the entity entity in dataConfig - say - pquery or preparedQuery and then pass the prepared statement and have values filled in by the actual queries for each row using a placeholder - like a ? or something else.
> I would probably start by hacking class JdbcDataSource to try a test but was wondering if anyone had experienced this or had any suggestions or if there is something in the works that I missed - I couldn't find any other bugs mentioning using prepared statements for performance.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.