DIH on sequence (or any type that supports ordering) possible?

[eks dev <ek...@yahoo.co.uk>]

I would appreciate some clarifications about DIH

I do not have reliable timestamp, but I do have atomic sequence that
only grows on inserts/changes.
You can understand it as a timestamp on some funky timezone not
related to wall clock time, it is integer type.

Is DIH keeping track of the MAX(committed timestamp) or it expects
timestamp in DB to be wall clock time?
If it expects wall clock timestamp, casting integer sequence value to
timestamp (like number of seconds since constant point in time) at
reading time would not work...

Ideally for my case, DIH should keep MAX(whatever_field_specified)...

Maybe in an Idea would be to modify DIH to support passing max(of the
specified field in dih config)  to the Lucene
IndexWriter.commit(Map<String,String> commitUserData)

Later, just read IndexReader.getCommitUserData() and pass it to SQL as
${last.committed.sequence}

This would have charming property, in a master slave setup, to
continue working after master fail over without touching anything....
every slave could overtake at any time

Second question is related to the delta queries as well. I know I have
no deletes/modifications in my DataSource, only additions.
Can I prevent DIH from trying to resolve deletes... my delta is fully
qualified by:
select * from source_table
where my_sequence > ${last.committed.sequence}

I imagine this step takes a lot of time to lookup every document ID in index?

Thanks in advance,
eks

Re: DIH on sequence (or any type that supports ordering) possible?

[eks dev <ek...@googlemail.com>]

Thanks Shawn, nice! I didn't notice you can pass more params all the
way to sql.

So you really do not care about DIH incremental facility,  you use it
just as vehicle to provide
- SQL import
- transactional commit to solr on updates...

But keeping DB/solr n sync is externalized (I am trying to find
simple/robust solution for this part as well...).

I am researching possibilities to get this information from lucene
index itself,  "what was the last document added?" , and than read
stored ID field from it to feed DIH query like yours

Should be easy question for solr/lucene to do, but I really do not
know simple and fast way...


cheers,
eks


On Sat, Aug 6, 2011 at 8:32 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 8/6/2011 8:49 AM, eks dev wrote:
>>
>> I would appreciate some clarifications about DIH
>>
>> I do not have reliable timestamp, but I do have atomic sequence that
>> only grows on inserts/changes.
>
> I use DIH, but I don't use the built-in timestamp facility at all.  I have
> an autoincrement field in a MySQL database that tells me what's new.  Here
> are the three queries I have defined in dih-config.xml:
>
>      query="
>        SELECT * FROM ${dataimporter.request.dataView}
>        WHERE (
>          (
>            did &gt; ${dataimporter.request.minDid}
>            AND did &lt;= ${dataimporter.request.maxDid}
>          )
>          ${dataimporter.request.extraWhere}
>        ) AND (crc32(did) % ${dataimporter.request.numShards})
>          IN (${dataimporter.request.modVal})
>        "
>      deltaImportQuery="
>        SELECT * FROM ${dataimporter.request.dataView}
>        WHERE (
>          (
>            did &gt; ${dataimporter.request.minDid}
>            AND did &lt;= ${dataimporter.request.maxDid}
>          )
>          ${dataimporter.request.extraWhere}
>        ) AND (crc32(did) % ${dataimporter.request.numShards})
>          IN (${dataimporter.request.modVal})
>        "
>      deltaQuery="SELECT 1 AS did"
>
> If you look carefully, you'll notice that query and deltaImportQuery are
> identical, and deltaQuery is just something that always returns a value.  I
> keep track of did (the primary key for both dih-config and the database) in
> my build system, passing in minDid and maxDid parameters on the DIH URL to
> tell it what to index.  I include more parameters to handle sharding and
> special situations.  I actually use a different field (with it's own unique
> MySQL index) as Solr's uniqueKey.
>
> Currently Solr does not support keeping track of arbitrary data, just the
> current timestamp ... but if you can track it outside of Solr and pass the
> appropriate parameters in with the full-import or delta-import request, you
> can do almost anything.
>
> This is on Solr 3.2, but I used a similar setup when I was running 1.4.1 as
> well.
>
> Shawn
>
>

Re: DIH on sequence (or any type that supports ordering) possible?

[Shawn Heisey <so...@elyograg.org>]

On 8/6/2011 8:49 AM, eks dev wrote:
> I would appreciate some clarifications about DIH
>
> I do not have reliable timestamp, but I do have atomic sequence that
> only grows on inserts/changes.

I use DIH, but I don't use the built-in timestamp facility at all.  I 
have an autoincrement field in a MySQL database that tells me what's 
new.  Here are the three queries I have defined in dih-config.xml:

       query="
         SELECT * FROM ${dataimporter.request.dataView}
         WHERE (
           (
             did > ${dataimporter.request.minDid}
             AND did <= ${dataimporter.request.maxDid}
           )
           ${dataimporter.request.extraWhere}
         ) AND (crc32(did) % ${dataimporter.request.numShards})
           IN (${dataimporter.request.modVal})
         "
       deltaImportQuery="
         SELECT * FROM ${dataimporter.request.dataView}
         WHERE (
           (
             did > ${dataimporter.request.minDid}
             AND did <= ${dataimporter.request.maxDid}
           )
           ${dataimporter.request.extraWhere}
         ) AND (crc32(did) % ${dataimporter.request.numShards})
           IN (${dataimporter.request.modVal})
         "
       deltaQuery="SELECT 1 AS did"

If you look carefully, you'll notice that query and deltaImportQuery are 
identical, and deltaQuery is just something that always returns a 
value.  I keep track of did (the primary key for both dih-config and the 
database) in my build system, passing in minDid and maxDid parameters on 
the DIH URL to tell it what to index.  I include more parameters to 
handle sharding and special situations.  I actually use a different 
field (with it's own unique MySQL index) as Solr's uniqueKey.

Currently Solr does not support keeping track of arbitrary data, just 
the current timestamp ... but if you can track it outside of Solr and 
pass the appropriate parameters in with the full-import or delta-import 
request, you can do almost anything.

This is on Solr 3.2, but I used a similar setup when I was running 1.4.1 
as well.

Shawn