Solr DataImportHandler

[Mark N <ni...@gmail.com>]

Is it possible to use solr DataImportHandler when that database fields are
not fixed ?  As per my findings we need to configure which table ( entity)
we will read the data and must match which fields in database will map to
fields in solr schema

Since in my case database fields could be dynamic , can DIH be helpful ?

please suggest


-- 
Nipen Mark

Re: Solr DataImportHandler

[Shawn Heisey <so...@elyograg.org>]

On 4/8/2010 7:05 AM, Shawn Heisey wrote:
>
> Here's what I'm using as the query in my latest config:

Actually, that was three separate queries:

query="SELECT * FROM ${dataimporter.request.dataTable} WHERE did > 
${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})"

deltaQuery="SELECT MAX(did) FROM ${dataimporter.request.dataTable}"

deltaImportQuery="SELECT * FROM ${dataimporter.request.dataTable} WHERE 
did > ${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})">

The deltaImportQuery is in fact identical to the regular query used for 
full-import.  The deltaQuery is just something related that returns 
quickly, the information is thrown away when it does a delta-import.

Shawn

Re: Solr DataImportHandler

[Shawn Heisey <so...@elyograg.org>]

On 4/8/2010 2:11 AM, Mark N wrote:
> Is it possible to use solr DataImportHandler when that database fields are
> not fixed ?  As per my findings we need to configure which table ( entity)
> we will read the data and must match which fields in database will map to
> fields in solr schema
>
> Since in my case database fields could be dynamic , can DIH be helpful ?
>
>    

You can pass variables into the DIH url that are then used in your query.

Here's what I'm using as the query in my latest config:

SELECT * FROM ${dataimporter.request.dataTable} WHERE did > 
${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})"
       deltaQuery="SELECT MAX(did) FROM ${dataimporter.request.dataTable}"
       deltaImportQuery="SELECT * FROM ${dataimporter.request.dataTable} 
WHERE did > ${dataimporter.request.minDid} AND did <= 
${dataimporter.request.maxDid} AND (did % 
${dataimporter.request.numShards}) IN (${dataimporter.request.modVal})

Then here is my URL template:

http://HOST:PORT/solr/CORE/dataimport?command=COMMAND&dataTable=DATATABLE&numShards=NUMSHARDS&modVal=MODVAL&minDid=MINDID&maxDid=MAXDID

And the perl data structure that holds the replacements for the 
uppercase parts:

$urlBits = {
   HOST => $cfg{'shards/inc.host1'},
   PORT => $cfg{'shards/inc.port'},
   MODVAL => $cfg{'shards/inc.modVal'},
   CORE => "live",
   COMMAND => "delta-import&commit=true&optimize=false",
   DATATABLE => $cfg{dataTable},
   NUMSHARDS => $cfg{numShards},
   MINDID => $cfg{maxDid},
   MAXDID => $dbMaxDid,
};