You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by clico <cl...@mairie-marseille.fr> on 2009/08/20 16:57:30 UTC
Remove data from index
Hello
I'm trying a way to do that :
I index a db query like
"select id from table_documents"
Some documents are updated or deleted from the data table.
Using DIH, I can indexe the updated document
But I want to remove from the index the documents that were removed in the
database.
How could I do this?
A way would be to delete in the index each time a doc is deleted in the
database but this is not possible
because I canot modify the code of my documents management tool
Thanks a lot
--
View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25063736.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remove data from index
Posted by Marc Sturlese <ma...@gmail.com>.
As far as I know you can not do that with DIH. What size is your index?
Probably the best you can do is index from scratch again with full-import.
clico wrote:
>
> I hope it could be a solution.
>
> But I think I understood that u can use deletePkQuery like this
>
> "select document_id from table_document where statusDeleted= 'Y'"
>
> In my case I have no status like "statusDeleted".
>
> The request I would like to write is
>
> "Delete from my solr Index the id that are no longer present in my
> table_document"
>
> With Lucene I had a way to do that :
> open IndexReader,
> for each lucene document : check in table_document and remove in lucene
> index if document is no longer present in the table
>
>
>
>
>
--
View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25063986.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Remove data from index
Posted by Fuad Efendi <fu...@efendi.ca>.
I had similar task but I simply used SolrJ client for that; I was forced to
keep track on deleted docs in table_document... now, I am using "delete by
query" (with SolrJ) - each document has a timestamp which I query...
For instance, you can update (allowOverwrite=true) whole SOLR index, but put
current timestamp on each document; then delete by query all exprired
documents
-Fuad
-----Original Message-----
From: clico [mailto:clico@mairie-marseille.fr]
Sent: August-21-09 5:58 AM
To: solr-user@lucene.apache.org
Subject: Re: Remove data from index
hossman wrote:
>
> : > The request I would like to write is
> : >
> : > "Delete from my solr Index the id that are no longer present in my
> : > table_document"
> : >
> : > With Lucene I had a way to do that :
> : > open IndexReader,
> : > for each lucene document : check in table_document and remove in
> lucene
> : > index if document is no longer present in the table
>
> you can still do that with Solr, you can even do it as a Solr plugin (I
> would suggest a RequestHandler that you hit after each DIH call) so
> you can reuse your existing code that deals directly with an IndexReader
> -- but you'd probably want to tweak it a bit to use the UpdateHandler from
> the SolrCore instead of deleting the doc direclty, that way it's logged
> properly and you can trigger a commit to make Solr aware you've modified
> the index.
>
> -Hoss
>
I read this answer and tried to understand the way to write new
RequestHandler
I think i begin to understand but don't see the way to implement it to
resolve my problem.
If I lit my CustomRequestHandler after each DIH how can it solve my delta pb
as the DIH
--
View this message in context:
http://www.nabble.com/Remove-data-from-index-tp25063736p25076138.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remove data from index
Posted by clico <cl...@mairie-marseille.fr>.
I don't understand this point.
--
View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25080669.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remove data from index
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
you can write a onImportEnd event handler also
On Fri, Aug 21, 2009 at 3:28 PM, clico<cl...@mairie-marseille.fr> wrote:
>
>
> hossman wrote:
>>
>> : > The request I would like to write is
>> : >
>> : > "Delete from my solr Index the id that are no longer present in my
>> : > table_document"
>> : >
>> : > With Lucene I had a way to do that :
>> : > open IndexReader,
>> : > for each lucene document : check in table_document and remove in
>> lucene
>> : > index if document is no longer present in the table
>>
>> you can still do that with Solr, you can even do it as a Solr plugin (I
>> would suggest a RequestHandler that you hit after each DIH call) so
>> you can reuse your existing code that deals directly with an IndexReader
>> -- but you'd probably want to tweak it a bit to use the UpdateHandler from
>> the SolrCore instead of deleting the doc direclty, that way it's logged
>> properly and you can trigger a commit to make Solr aware you've modified
>> the index.
>>
>> -Hoss
>>
>
> I read this answer and tried to understand the way to write new
> RequestHandler
> I think i begin to understand but don't see the way to implement it to
> resolve my problem.
>
> If I lit my CustomRequestHandler after each DIH how can it solve my delta pb
> as the DIH
>
>
> --
> View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25076138.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Remove data from index
Posted by clico <cl...@mairie-marseille.fr>.
hossman wrote:
>
> : > The request I would like to write is
> : >
> : > "Delete from my solr Index the id that are no longer present in my
> : > table_document"
> : >
> : > With Lucene I had a way to do that :
> : > open IndexReader,
> : > for each lucene document : check in table_document and remove in
> lucene
> : > index if document is no longer present in the table
>
> you can still do that with Solr, you can even do it as a Solr plugin (I
> would suggest a RequestHandler that you hit after each DIH call) so
> you can reuse your existing code that deals directly with an IndexReader
> -- but you'd probably want to tweak it a bit to use the UpdateHandler from
> the SolrCore instead of deleting the doc direclty, that way it's logged
> properly and you can trigger a commit to make Solr aware you've modified
> the index.
>
> -Hoss
>
I read this answer and tried to understand the way to write new
RequestHandler
I think i begin to understand but don't see the way to implement it to
resolve my problem.
If I lit my CustomRequestHandler after each DIH how can it solve my delta pb
as the DIH
--
View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25076138.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remove data from index
Posted by Chris Hostetter <ho...@fucit.org>.
: > The request I would like to write is
: >
: > "Delete from my solr Index the id that are no longer present in my
: > table_document"
: >
: > With Lucene I had a way to do that :
: > open IndexReader,
: > for each lucene document : check in table_document and remove in lucene
: > index if document is no longer present in the table
you can still do that with Solr, you can even do it as a Solr plugin (I
would suggest a RequestHandler that you hit after each DIH call) so
you can reuse your existing code that deals directly with an IndexReader
-- but you'd probably want to tweak it a bit to use the UpdateHandler from
the SolrCore instead of deleting the doc direclty, that way it's logged
properly and you can trigger a commit to make Solr aware you've modified
the index.
-Hoss
Re: Remove data from index
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
On Thu, Aug 20, 2009 at 8:39 PM, clico<cl...@mairie-marseille.fr> wrote:
>
> I hope it could be a solution.
>
> But I think I understood that u can use deletePkQuery like this
>
> "select document_id from table_document where statusDeleted= 'Y'"
>
> In my case I have no status like "statusDeleted".
I don't think there is a straight solution w/o doing a full-import
>
> The request I would like to write is
>
> "Delete from my solr Index the id that are no longer present in my
> table_document"
>
> With Lucene I had a way to do that :
> open IndexReader,
> for each lucene document : check in table_document and remove in lucene
> index if document is no longer present in the table
>
>
>
>
> --
> View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25063965.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: Remove data from index
Posted by Constantijn Visinescu <ba...@gmail.com>.
You could write a solr query that queries *:* and only returns the id field
and then throw out all the IDs from "select id from databaseTable"
and then run a delete query for all the IDs that are left after wards.
However you'd have to write a seperate program/script to do this i think as
the DIH won't be able to do this my itself.
On Thu, Aug 20, 2009 at 5:09 PM, clico <cl...@mairie-marseille.fr> wrote:
>
> I hope it could be a solution.
>
> But I think I understood that u can use deletePkQuery like this
>
> "select document_id from table_document where statusDeleted= 'Y'"
>
> In my case I have no status like "statusDeleted".
>
> The request I would like to write is
>
> "Delete from my solr Index the id that are no longer present in my
> table_document"
>
> With Lucene I had a way to do that :
> open IndexReader,
> for each lucene document : check in table_document and remove in lucene
> index if document is no longer present in the table
>
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Remove-data-from-index-tp25063736p25063965.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: Remove data from index
Posted by clico <cl...@mairie-marseille.fr>.
I hope it could be a solution.
But I think I understood that u can use deletePkQuery like this
"select document_id from table_document where statusDeleted= 'Y'"
In my case I have no status like "statusDeleted".
The request I would like to write is
"Delete from my solr Index the id that are no longer present in my
table_document"
With Lucene I had a way to do that :
open IndexReader,
for each lucene document : check in table_document and remove in lucene
index if document is no longer present in the table
--
View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25063965.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Remove data from index
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
did you see the deletedPkQuery?
On Thu, Aug 20, 2009 at 8:27 PM, clico<cl...@mairie-marseille.fr> wrote:
>
> Hello
>
> I'm trying a way to do that :
>
> I index a db query like
> "select id from table_documents"
>
> Some documents are updated or deleted from the data table.
>
>
> Using DIH, I can indexe the updated document
>
> But I want to remove from the index the documents that were removed in the
> database.
>
> How could I do this?
> A way would be to delete in the index each time a doc is deleted in the
> database but this is not possible
> because I canot modify the code of my documents management tool
>
> Thanks a lot
>
>
>
>
> --
> View this message in context: http://www.nabble.com/Remove-data-from-index-tp25063736p25063736.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com