You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Olivier Poitrey (JIRA)" <ji...@apache.org> on 2008/06/03 23:58:45 UTC
[jira] Commented: (SOLR-469) Data Import RequestHandler
[ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602089#action_12602089 ]
Olivier Poitrey commented on SOLR-469:
--------------------------------------
Paul,
The current version of the code seems not to allow the construction pk="forum.forumId" you're talking about. I did a small patch to make it possible, I don't know if it's the correct way to do it but it works well for me. Here is the patch:
{noformat}
--- a/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java
+++ b/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java
@@ -124,7 +124,9 @@ public class SqlEntityProcessor extends EntityProcessorBase {
sb.append(" and ");
first = false;
}
- Object val = resolver.resolve(primaryKeys[i]);
+ // Only send the field part of the pk when pk includes the table ref
+ String[] pkParts = primaryKeys[i].split("\\.");
+ Object val = resolver.resolve(pkParts[pkParts.length - 1]);
sb.append(primaryKeys[i]).append(" = ");
if (val instanceof Number) {
sb.append(val.toString());
{noformat}
Hope that helps.
> Data Import RequestHandler
> --------------------------
>
> Key: SOLR-469
> URL: https://issues.apache.org/jira/browse/SOLR-469
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 1.3
> Reporter: Noble Paul
> Assignee: Grant Ingersoll
> Fix For: 1.3
>
> Attachments: SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
>
>
> We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103).
> The way it works is as follows.
> * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema
> - It also takes in a properties file for the data source configuraution
> * Given the configuration it can also generate the solr schema.xml
> * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import
> - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration)
> - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables)
> * It provides a admin page
> - where we can schedule it to be run automatically at regular intervals
> - It shows the status of the Handler (idle, full-import, delta-import)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SOLR-469) Data Import RequestHandler
Posted by Noble Paul നോബിള് नोब्ळ् <no...@gmail.com>.
hi Olivier,
I guess this is the same problem reported by Moser. Your fix is more
or less fine. We have made a fix that is similar to what you have
done. The next patch will have these fixes also.if you wish we will
post the java file separately.
We are just waiting for the first patch to get committed so that these
can be bug fixes on that.
--Noble
On Wed, Jun 4, 2008 at 3:28 AM, Olivier Poitrey (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/jira/browse/SOLR-469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602089#action_12602089 ]
>
> Olivier Poitrey commented on SOLR-469:
> --------------------------------------
>
> Paul,
>
> The current version of the code seems not to allow the construction pk="forum.forumId" you're talking about. I did a small patch to make it possible, I don't know if it's the correct way to do it but it works well for me. Here is the patch:
>
> {noformat}
> --- a/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java
> +++ b/contrib/dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/SqlEntityProcessor.java
> @@ -124,7 +124,9 @@ public class SqlEntityProcessor extends EntityProcessorBase {
> sb.append(" and ");
> first = false;
> }
> - Object val = resolver.resolve(primaryKeys[i]);
> + // Only send the field part of the pk when pk includes the table ref
> + String[] pkParts = primaryKeys[i].split("\\.");
> + Object val = resolver.resolve(pkParts[pkParts.length - 1]);
> sb.append(primaryKeys[i]).append(" = ");
> if (val instanceof Number) {
> sb.append(val.toString());
> {noformat}
>
> Hope that helps.
>
>> Data Import RequestHandler
>> --------------------------
>>
>> Key: SOLR-469
>> URL: https://issues.apache.org/jira/browse/SOLR-469
>> Project: Solr
>> Issue Type: New Feature
>> Components: update
>> Affects Versions: 1.3
>> Reporter: Noble Paul
>> Assignee: Grant Ingersoll
>> Fix For: 1.3
>>
>> Attachments: SOLR-469-contrib.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch, SOLR-469.patch
>>
>>
>> We need a RequestHandler Which can import data from a DB or other dataSources into the Solr index .Think of it as an advanced form of SqlUpload Plugin (SOLR-103).
>> The way it works is as follows.
>> * Provide a configuration file (xml) to the Handler which takes in the necessary SQL queries and mappings to a solr schema
>> - It also takes in a properties file for the data source configuraution
>> * Given the configuration it can also generate the solr schema.xml
>> * It is registered as a RequestHandler which can take two commands do-full-import, do-delta-import
>> - do-full-import - dumps all the data from the Database into the index (based on the SQL query in configuration)
>> - do-delta-import - dumps all the data that has changed since last import. (We assume a modified-timestamp column in tables)
>> * It provides a admin page
>> - where we can schedule it to be run automatically at regular intervals
>> - It shows the status of the Handler (idle, full-import, delta-import)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
--Noble Paul