You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by bbarani <bb...@gmail.com> on 2010/10/20 16:51:47 UTC
How to delete a SOLR document if that particular data doesnt exist
in DB?
Hi,
I have a very common question but couldnt find any post related to my
question in this forum,
I am currently initiating a full import each week but the data that have
been deleted in the source is not update in my document as I am using
clean=false.
We are indexing multiple data by data types hence cant delete the index and
do a complete re-indexing each week also we want to delete the orphan solr
documents (for which the data is not present in back end DB) on a daily
basis.
Now my question is.. Is there a way I can use preImportDeleteQuery to delete
the documents from SOLR for which the data doesnt exist in back end db? I
dont have anything called delete status in DB, instead I need to get all the
UID's from SOLR document and compare it with all the UID's in back end and
delete the data from SOLR document for the UID's which is not present in DB.
Any suggestion / ideas would be of great help.
Note: Currently I have developed a simple program which will fetch the UID's
from SOLR document and then connect to backend DB to check the orphan UID's
and delete the documents from SOLR index corresponding to orphan UID's. I
just dont want to re-invent the wheel if this feature is already present in
SOLR as I need to do more testing in terms of performance / scalability for
my program..
Thanks,
Barani
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1739222.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by Mike Sokolov <so...@ifactory.com>.
Since you are performing a complete reload of all of your data, I don't
understand why you can't create a new core, load your new data, swap
your application to look at the new core, and then erase the old one, if
you want.
Even so, you could track the timestamps on all your documents, which
will be updated when you update the content. Then when you're done you
could delete anything with a timestamp prior to the time you started the
latest import.
-Mike
On 10/20/2010 11:59 AM, bbarani wrote:
> ironicnet,
>
> Thanks for your reply.
>
> We actually use virtual DB modelling tool to fetch the data from various
> sources during run time hence we dont have any control over the source.
>
> We consolidate the data from more than one source and index the consolidated
> data using SOLR. We dont have any kind of update / access rights to source
> data.
>
> Thanks.
> Barani
>
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by Ezequiel Calderara <ez...@gmail.com>.
Also you can set an expiration policy maybe, and delete files that expire
after some time and aren't older than other... but i don't know if you can
iterate over the existing ids...
On Wed, Oct 20, 2010 at 1:34 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 10/20/2010 9:59 AM, bbarani wrote:
>
>> We actually use virtual DB modelling tool to fetch the data from various
>> sources during run time hence we dont have any control over the source.
>>
>> We consolidate the data from more than one source and index the
>> consolidated
>> data using SOLR. We dont have any kind of update / access rights to source
>> data.
>>
>
> It seems likely that those who are in control of the data sources would be
> maintaining some kind of delete log, and that they should be able to make
> those logs available to you.
>
> For my index, the data comes from a MySQL database. When a delete is done
> at the database level, a database trigger records the old information to a
> main delete log table, as well as a separate table for the search system.
> The build system uses that separate table to run deletes every ten minutes
> and keeps it trimmed to 48 hours of delete history.
>
>
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/20/2010 9:59 AM, bbarani wrote:
> We actually use virtual DB modelling tool to fetch the data from various
> sources during run time hence we dont have any control over the source.
>
> We consolidate the data from more than one source and index the consolidated
> data using SOLR. We dont have any kind of update / access rights to source
> data.
It seems likely that those who are in control of the data sources would
be maintaining some kind of delete log, and that they should be able to
make those logs available to you.
For my index, the data comes from a MySQL database. When a delete is
done at the database level, a database trigger records the old
information to a main delete log table, as well as a separate table for
the search system. The build system uses that separate table to run
deletes every ten minutes and keeps it trimmed to 48 hours of delete
history.
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by bbarani <bb...@gmail.com>.
ironicnet,
Thanks for your reply.
We actually use virtual DB modelling tool to fetch the data from various
sources during run time hence we dont have any control over the source.
We consolidate the data from more than one source and index the consolidated
data using SOLR. We dont have any kind of update / access rights to source
data.
Thanks.
Barani
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1739642.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by Ezequiel Calderara <ez...@gmail.com>.
Can't you in each delete of that data, save the ids in other table?
And then process those ids against solr to delete them?
On Wed, Oct 20, 2010 at 11:51 AM, bbarani <bb...@gmail.com> wrote:
>
> Hi,
>
> I have a very common question but couldnt find any post related to my
> question in this forum,
>
> I am currently initiating a full import each week but the data that have
> been deleted in the source is not update in my document as I am using
> clean=false.
>
> We are indexing multiple data by data types hence cant delete the index and
> do a complete re-indexing each week also we want to delete the orphan solr
> documents (for which the data is not present in back end DB) on a daily
> basis.
>
> Now my question is.. Is there a way I can use preImportDeleteQuery to
> delete
> the documents from SOLR for which the data doesnt exist in back end db? I
> dont have anything called delete status in DB, instead I need to get all
> the
> UID's from SOLR document and compare it with all the UID's in back end and
> delete the data from SOLR document for the UID's which is not present in
> DB.
>
> Any suggestion / ideas would be of great help.
>
> Note: Currently I have developed a simple program which will fetch the
> UID's
> from SOLR document and then connect to backend DB to check the orphan UID's
> and delete the documents from SOLR index corresponding to orphan UID's. I
> just dont want to re-invent the wheel if this feature is already present in
> SOLR as I need to do more testing in terms of performance / scalability for
> my program..
>
> Thanks,
> Barani
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1739222.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
______
Ezequiel.
Http://www.ironicnet.com
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by bbarani <bb...@gmail.com>.
Thanks a lot for all your replies.
I finally wrote a program which will fetch and store all the UID from source
(DB) in one list and fetch and store all the UID from SOLR document in
another list.
Next using the binarySearch method of collection I was able to filter out
the list of UID's that are not present in SOLR UID list with that of DB UID
list and passed those UID for deletion using deletebyQuery.
It took under 7 minutes to compare 2 list with over 3 million records (in
each list) and delete the orphan documents from SOLR index.
Again thanks a lot for all your replies.
Thanks,
Barani
--
View this message in context: http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1761093.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by ben boggess <be...@gmail.com>.
> Now my question is.. Is there a way I can use preImportDeleteQuery to
> delete
> the documents from SOLR for which the data doesnt exist in back end db? I
> dont have anything called delete status in DB, instead I need to get all
> the
> UID's from SOLR document and compare it with all the UID's in back end and
> delete the data from SOLR document for the UID's which is not present in
> DB.
I've done something like this with raw Lucene and I'm not sure how or if you
could do it with Solr as I'm relatively new to it.
We stored a timestamp for when we started to import and stored an update
timestamp field for every document added to the index. After the data
import, we did a delete by query that matched all documents with a timestamp
older than when we started. The assumption being that if we didn't update
the timestamp during the load, then the record must have been deleted from
the database.
Hope this helps.
Ben
On Wed, Oct 20, 2010 at 8:05 PM, Erick Erickson <er...@gmail.com>wrote:
> <<<We are indexing multiple data by data types hence cant delete the index
> and
> do a complete re-indexing each week also we want to delete the orphan solr
> documents (for which the data is not present in back end DB) on a daily
> basis.>>>
>
> Can you make delete by query work? Something like delete all Solr docs of
> a certain type and do a full re-index of just that type?
>
> I have no idea whether this is practical or not....
>
> But your solution also works. There's really no way Solr #can# know about
> deleted database records, especially since the <uniqueKey> field is
> completely
> arbitrarily defined.
>
> Best
> Erick
>
> On Wed, Oct 20, 2010 at 10:51 AM, bbarani <bb...@gmail.com> wrote:
>
> >
> > Hi,
> >
> > I have a very common question but couldnt find any post related to my
> > question in this forum,
> >
> > I am currently initiating a full import each week but the data that have
> > been deleted in the source is not update in my document as I am using
> > clean=false.
> >
> > We are indexing multiple data by data types hence cant delete the index
> and
> > do a complete re-indexing each week also we want to delete the orphan
> solr
> > documents (for which the data is not present in back end DB) on a daily
> > basis.
> >
> > Now my question is.. Is there a way I can use preImportDeleteQuery to
> > delete
> > the documents from SOLR for which the data doesnt exist in back end db? I
> > dont have anything called delete status in DB, instead I need to get all
> > the
> > UID's from SOLR document and compare it with all the UID's in back end
> and
> > delete the data from SOLR document for the UID's which is not present in
> > DB.
> >
> > Any suggestion / ideas would be of great help.
> >
> > Note: Currently I have developed a simple program which will fetch the
> > UID's
> > from SOLR document and then connect to backend DB to check the orphan
> UID's
> > and delete the documents from SOLR index corresponding to orphan UID's. I
> > just dont want to re-invent the wheel if this feature is already present
> in
> > SOLR as I need to do more testing in terms of performance / scalability
> for
> > my program..
> >
> > Thanks,
> > Barani
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1739222.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: How to delete a SOLR document if that particular data doesnt
exist in DB?
Posted by Erick Erickson <er...@gmail.com>.
<<<We are indexing multiple data by data types hence cant delete the index
and
do a complete re-indexing each week also we want to delete the orphan solr
documents (for which the data is not present in back end DB) on a daily
basis.>>>
Can you make delete by query work? Something like delete all Solr docs of
a certain type and do a full re-index of just that type?
I have no idea whether this is practical or not....
But your solution also works. There's really no way Solr #can# know about
deleted database records, especially since the <uniqueKey> field is
completely
arbitrarily defined.
Best
Erick
On Wed, Oct 20, 2010 at 10:51 AM, bbarani <bb...@gmail.com> wrote:
>
> Hi,
>
> I have a very common question but couldnt find any post related to my
> question in this forum,
>
> I am currently initiating a full import each week but the data that have
> been deleted in the source is not update in my document as I am using
> clean=false.
>
> We are indexing multiple data by data types hence cant delete the index and
> do a complete re-indexing each week also we want to delete the orphan solr
> documents (for which the data is not present in back end DB) on a daily
> basis.
>
> Now my question is.. Is there a way I can use preImportDeleteQuery to
> delete
> the documents from SOLR for which the data doesnt exist in back end db? I
> dont have anything called delete status in DB, instead I need to get all
> the
> UID's from SOLR document and compare it with all the UID's in back end and
> delete the data from SOLR document for the UID's which is not present in
> DB.
>
> Any suggestion / ideas would be of great help.
>
> Note: Currently I have developed a simple program which will fetch the
> UID's
> from SOLR document and then connect to backend DB to check the orphan UID's
> and delete the documents from SOLR index corresponding to orphan UID's. I
> just dont want to re-invent the wheel if this feature is already present in
> SOLR as I need to do more testing in terms of performance / scalability for
> my program..
>
> Thanks,
> Barani
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/How-to-delete-a-SOLR-document-if-that-particular-data-doesnt-exist-in-DB-tp1739222p1739222.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>