You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Mark Ellul <ma...@catalystic.com> on 2009/11/16 18:42:44 UTC
$DeleteDocbyQuery in solr 1.4 is not working
Hi,
I have added a deleted field in my database, and am using the
Dataimporthandler to add rows to the index...
I am using solr 1.4
I have added my the deleted field to the query and the RegexTransformer...
and the field definition below
<field column="$deleteDocByQuery"
regex="^true$"
replaceWith="id:${List.id}" sourceColName="deleted"/>
When I run the deltaImport command... I see the below output
INFO: [] webapp=/solr path=/dataimport
params={command=delta-import&debug=true&expungeDeletes=true} status=0
QTime=1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
INFO: Starting Delta Import
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
readIndexerProperties
INFO: Read dataimport.properties
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Starting delta collection.
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Running ModifiedRowKey() for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Creating a connection for entity List with URL:
jdbc:postgresql://localhost:5432/tlists
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
call
INFO: Time taken for getConnection(): 4
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
collectDelta
INFO: Completed parentDeltaQuery for Entity: List
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
deleteByQuery
INFO: Deleting documents from Solr with query: id:api__list__365522
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
_bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1257863009839
Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
doDelta
INFO: Delta Import completed successfully
It says its deleting the document... but when I do the search its still
showing up....
Any Ideas?
Regards
Mark
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by cpmoser <cp...@gmail.com>.
Ok, I think I figured out what might be happening. It appears that the
DataImporter issues the commit command without the expungeDeletes option set
to true (default in 1.4 for a commit command is for expungeDeletes to be set
to false). You can get around this by issuing the commit command manually:
curl
"http://localhost:8983/solr/dataimport?command=delta-import&commit=false&optimize=false"
curl "http://localhost:8983/solr/update" --data-binary '<commit
expungeDeletes="true" />' -H "Content-type:text/xml; charset=utf-8"
That fixed the issue for me, although ideally the DataImportHandler should
run with the expungeDeletes option set to true, so that this could all
happen in one command without having to wait for the DIH to finish before
issuing the commit command.
Is there a way to set the expungeDeletes option for the commit command of
the DataImportHandler, so that delta imports with deletes can happen
automatically?
Mark.El wrote:
>
> Thanks I will look into it!
>
>
--
View this message in context: http://old.nabble.com/%24DeleteDocbyQuery-in-solr-1.4-is-not-working-tp26376265p26583260.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Mark Ellul <ma...@catalystic.com>.
Thanks I will look into it!
On Fri, Nov 27, 2009 at 11:34 PM, cpmoser <cp...@gmail.com> wrote:
>
> Hi I just recently (well today actually) ran into the same issue, and a
> Google search led me here. Something in the log that clued me in to my
> issue was this:
>
> Nov 27, 2009 1:27:05 PM org.apache.solr.core.SolrDeletionPolicy onInit
>
> I was expecting docs to be removed on commit (which normally happens after
> a
> dataimport), so I thought something was broken.
>
> However, when I restarted Solr, the docs that should have been deleted were
> actually deleted. There is more discussion about the SolrDeletionPolicy
> http://issues.apache.org/jira/browse/SOLR-617 here . I haven't read
> enough
> to know how to add an onCommit deletion policy to the Solr config yet, and
> don't know if you're running into the same issue, but hope this helps.
>
>
> Mark.El wrote:
> >
> > Thanks Otis... I remember that one!
> >
> > It still did not remove the document! So obviously its something else
> > thats
> > happening.
> >
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/%24DeleteDocbyQuery-in-solr-1.4-is-not-working-tp26376265p26545394.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by cpmoser <cp...@gmail.com>.
Hi I just recently (well today actually) ran into the same issue, and a
Google search led me here. Something in the log that clued me in to my
issue was this:
Nov 27, 2009 1:27:05 PM org.apache.solr.core.SolrDeletionPolicy onInit
I was expecting docs to be removed on commit (which normally happens after a
dataimport), so I thought something was broken.
However, when I restarted Solr, the docs that should have been deleted were
actually deleted. There is more discussion about the SolrDeletionPolicy
http://issues.apache.org/jira/browse/SOLR-617 here . I haven't read enough
to know how to add an onCommit deletion policy to the Solr config yet, and
don't know if you're running into the same issue, but hope this helps.
Mark.El wrote:
>
> Thanks Otis... I remember that one!
>
> It still did not remove the document! So obviously its something else
> thats
> happening.
>
>
>
--
View this message in context: http://old.nabble.com/%24DeleteDocbyQuery-in-solr-1.4-is-not-working-tp26376265p26545394.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Mark Ellul <ma...@catalystic.com>.
Thanks Otis... I remember that one!
It still did not remove the document! So obviously its something else thats
happening.
On Tue, Nov 17, 2009 at 10:47 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Mark,
>
> http://localhost:8983/solr/update?stream.body=%3Ccommit/%3E
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
>
> ----- Original Message ----
> > From: Mark Ellul <ma...@catalystic.com>
> > To: solr-user@lucene.apache.org; noble.paul@gmail.com
> > Sent: Tue, November 17, 2009 9:22:13 AM
> > Subject: Re: $DeleteDocbyQuery in solr 1.4 is not working
> >
> > Hi Noble,
> >
> > I have updated my entity specs, by having a separate entity for
> > selecting rows which are not deleted for and ones that are deleted, so
> > I am sure now that the document is not getting added in the same
> > import.
> >
> > I read in the tutorial that the deletes are not taken out until the
> > commit is done. Is there a way I can force a commit? I don't have the
> > post.jar on the machine in question.
> >
> > So whats next, I can now safely say that the documents are not getting
> > added again, and they should be getting deleted... but they not,
> > unless its not committing and I have to do it manually.
> >
> > Any other ideas?
> >
> > Thanks for your help so far!
> >
> > Regards
> >
> > Mark
> >
> > On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > > why don't you add a new timestamp field . you can use the
> > > TemplateTransformer with the formatDate() function
> > >
> > > On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
> > >> Hi Noble,
> > >>
> > >> Excellent Question... should the field that does the deleting be in a
> > >> different entity to the one that does the addition and updating?
> > >>
> > >> If so that could be the issue, I have the field that does the
> > >> DeleteByQuery command inside of the entity that does the adding.
> > >>
> > >> Is there some kind of document metadata where the create date and
> > >> update date is show?
> > >>
> > >> How would I see this meta data if it exists?
> > >>
> > >> Regards
> > >>
> > >> Mark
> > >>
> > >> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > >>> The question is, did your new delta-import created the doc again?
> > >>>
> > >>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
> > >>>> The doc already existed before the delta-import has been run.
> > >>>>
> > >>>> And it exists afterwards... even though it says its deleting it.
> > >>>>
> > >>>> Any ideas of what I can try?
> > >>>>
> > >>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > >>>>> are you sure that the doc w/ the same id was not created after
> that?
> > >>>>>
> > >>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul
> > >>>>> wrote:
> > >>>>>> Hi,
> > >>>>>>
> > >>>>>> I have added a deleted field in my database, and am using the
> > >>>>>> Dataimporthandler to add rows to the index...
> > >>>>>>
> > >>>>>> I am using solr 1.4
> > >>>>>>
> > >>>>>> I have added my the deleted field to the query and the
> > >>>>>> RegexTransformer...
> > >>>>>> and the field definition below
> > >>>>>>
> > >>>>>>
> > >>>>>> regex="^true$"
> > >>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
> > >>>>>>
> > >>>>>> When I run the deltaImport command... I see the below output
> > >>>>>>
> > >>>>>> INFO: [] webapp=/solr path=/dataimport
> > >>>>>> params={command=delta-import&debug=true&expungeDeletes=true}
> status=0
> > >>>>>> QTime=1
> > >>>>>> Nov 16, 2009 5:29:10 PM
> > >>>>>> org.apache.solr.handler.dataimport.DataImporter
> > >>>>>> doDeltaImport
> > >>>>>> INFO: Starting Delta Import
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.SolrWriter
> > >>>>>> readIndexerProperties
> > >>>>>> INFO: Read dataimport.properties
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> doDelta
> > >>>>>> INFO: Starting delta collection.
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> collectDelta
> > >>>>>> INFO: Running ModifiedRowKey() for Entity: List
> > >>>>>> Nov 16, 2009 5:29:10 PM
> > >>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
> > >>>>>> call
> > >>>>>> INFO: Creating a connection for entity List with URL:
> > >>>>>> jdbc:postgresql://localhost:5432/tlists
> > >>>>>> Nov 16, 2009 5:29:10 PM
> > >>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
> > >>>>>> call
> > >>>>>> INFO: Time taken for getConnection(): 4
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> collectDelta
> > >>>>>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> collectDelta
> > >>>>>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> collectDelta
> > >>>>>> INFO: Completed parentDeltaQuery for Entity: List
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.SolrWriter
> > >>>>>> deleteByQuery
> > >>>>>> INFO: Deleting documents from Solr with query:
> id:api__list__365522
> > >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
> onInit
> > >>>>>> INFO: SolrDeletionPolicy.onInit: commits:num=1
> > >>>>>>
> >
> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
> > >>>>>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
> > >>>>>> _bg.frq]
> > >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
> > >>>>>> updateCommits
> > >>>>>> INFO: newest commit = 1257863009839
> > >>>>>> Nov 16, 2009 5:29:10 PM
> org.apache.solr.handler.dataimport.DocBuilder
> > >>>>>> doDelta
> > >>>>>> INFO: Delta Import completed successfully
> > >>>>>>
> > >>>>>> It says its deleting the document... but when I do the search its
> > >>>>>> still
> > >>>>>> showing up....
> > >>>>>>
> > >>>>>> Any Ideas?
> > >>>>>>
> > >>>>>> Regards
> > >>>>>>
> > >>>>>> Mark
> > >>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> -----------------------------------------------------
> > >>>>> Noble Paul | Principal Engineer| AOL | http://aol.com
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -----------------------------------------------------
> > >>> Noble Paul | Principal Engineer| AOL | http://aol.com
> > >>>
> > >>
> > >
> > >
> > >
> > > --
> > > -----------------------------------------------------
> > > Noble Paul | Principal Engineer| AOL | http://aol.com
> > >
>
>
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Mark,
http://localhost:8983/solr/update?stream.body=%3Ccommit/%3E
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Mark Ellul <ma...@catalystic.com>
> To: solr-user@lucene.apache.org; noble.paul@gmail.com
> Sent: Tue, November 17, 2009 9:22:13 AM
> Subject: Re: $DeleteDocbyQuery in solr 1.4 is not working
>
> Hi Noble,
>
> I have updated my entity specs, by having a separate entity for
> selecting rows which are not deleted for and ones that are deleted, so
> I am sure now that the document is not getting added in the same
> import.
>
> I read in the tutorial that the deletes are not taken out until the
> commit is done. Is there a way I can force a commit? I don't have the
> post.jar on the machine in question.
>
> So whats next, I can now safely say that the documents are not getting
> added again, and they should be getting deleted... but they not,
> unless its not committing and I have to do it manually.
>
> Any other ideas?
>
> Thanks for your help so far!
>
> Regards
>
> Mark
>
> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> > why don't you add a new timestamp field . you can use the
> > TemplateTransformer with the formatDate() function
> >
> > On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul wrote:
> >> Hi Noble,
> >>
> >> Excellent Question... should the field that does the deleting be in a
> >> different entity to the one that does the addition and updating?
> >>
> >> If so that could be the issue, I have the field that does the
> >> DeleteByQuery command inside of the entity that does the adding.
> >>
> >> Is there some kind of document metadata where the create date and
> >> update date is show?
> >>
> >> How would I see this meta data if it exists?
> >>
> >> Regards
> >>
> >> Mark
> >>
> >> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> >>> The question is, did your new delta-import created the doc again?
> >>>
> >>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul wrote:
> >>>> The doc already existed before the delta-import has been run.
> >>>>
> >>>> And it exists afterwards... even though it says its deleting it.
> >>>>
> >>>> Any ideas of what I can try?
> >>>>
> >>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् wrote:
> >>>>> are you sure that the doc w/ the same id was not created after that?
> >>>>>
> >>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul
> >>>>> wrote:
> >>>>>> Hi,
> >>>>>>
> >>>>>> I have added a deleted field in my database, and am using the
> >>>>>> Dataimporthandler to add rows to the index...
> >>>>>>
> >>>>>> I am using solr 1.4
> >>>>>>
> >>>>>> I have added my the deleted field to the query and the
> >>>>>> RegexTransformer...
> >>>>>> and the field definition below
> >>>>>>
> >>>>>>
> >>>>>> regex="^true$"
> >>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
> >>>>>>
> >>>>>> When I run the deltaImport command... I see the below output
> >>>>>>
> >>>>>> INFO: [] webapp=/solr path=/dataimport
> >>>>>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
> >>>>>> QTime=1
> >>>>>> Nov 16, 2009 5:29:10 PM
> >>>>>> org.apache.solr.handler.dataimport.DataImporter
> >>>>>> doDeltaImport
> >>>>>> INFO: Starting Delta Import
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
> >>>>>> readIndexerProperties
> >>>>>> INFO: Read dataimport.properties
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> doDelta
> >>>>>> INFO: Starting delta collection.
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> collectDelta
> >>>>>> INFO: Running ModifiedRowKey() for Entity: List
> >>>>>> Nov 16, 2009 5:29:10 PM
> >>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
> >>>>>> call
> >>>>>> INFO: Creating a connection for entity List with URL:
> >>>>>> jdbc:postgresql://localhost:5432/tlists
> >>>>>> Nov 16, 2009 5:29:10 PM
> >>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
> >>>>>> call
> >>>>>> INFO: Time taken for getConnection(): 4
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> collectDelta
> >>>>>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> collectDelta
> >>>>>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> collectDelta
> >>>>>> INFO: Completed parentDeltaQuery for Entity: List
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
> >>>>>> deleteByQuery
> >>>>>> INFO: Deleting documents from Solr with query: id:api__list__365522
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
> >>>>>> INFO: SolrDeletionPolicy.onInit: commits:num=1
> >>>>>>
> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
> >>>>>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
> >>>>>> _bg.frq]
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
> >>>>>> updateCommits
> >>>>>> INFO: newest commit = 1257863009839
> >>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> >>>>>> doDelta
> >>>>>> INFO: Delta Import completed successfully
> >>>>>>
> >>>>>> It says its deleting the document... but when I do the search its
> >>>>>> still
> >>>>>> showing up....
> >>>>>>
> >>>>>> Any Ideas?
> >>>>>>
> >>>>>> Regards
> >>>>>>
> >>>>>> Mark
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> -----------------------------------------------------
> >>>>> Noble Paul | Principal Engineer| AOL | http://aol.com
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> -----------------------------------------------------
> >>> Noble Paul | Principal Engineer| AOL | http://aol.com
> >>>
> >>
> >
> >
> >
> > --
> > -----------------------------------------------------
> > Noble Paul | Principal Engineer| AOL | http://aol.com
> >
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Mark Ellul <ma...@catalystic.com>.
Hi Noble,
I have updated my entity specs, by having a separate entity for
selecting rows which are not deleted for and ones that are deleted, so
I am sure now that the document is not getting added in the same
import.
I read in the tutorial that the deletes are not taken out until the
commit is done. Is there a way I can force a commit? I don't have the
post.jar on the machine in question.
So whats next, I can now safely say that the documents are not getting
added again, and they should be getting deleted... but they not,
unless its not committing and I have to do it manually.
Any other ideas?
Thanks for your help so far!
Regards
Mark
On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
> why don't you add a new timestamp field . you can use the
> TemplateTransformer with the formatDate() function
>
> On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul <ma...@catalystic.com> wrote:
>> Hi Noble,
>>
>> Excellent Question... should the field that does the deleting be in a
>> different entity to the one that does the addition and updating?
>>
>> If so that could be the issue, I have the field that does the
>> DeleteByQuery command inside of the entity that does the adding.
>>
>> Is there some kind of document metadata where the create date and
>> update date is show?
>>
>> How would I see this meta data if it exists?
>>
>> Regards
>>
>> Mark
>>
>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
>>> The question is, did your new delta-import created the doc again?
>>>
>>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul <ma...@catalystic.com> wrote:
>>>> The doc already existed before the delta-import has been run.
>>>>
>>>> And it exists afterwards... even though it says its deleting it.
>>>>
>>>> Any ideas of what I can try?
>>>>
>>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
>>>>> are you sure that the doc w/ the same id was not created after that?
>>>>>
>>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul <ma...@catalystic.com>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have added a deleted field in my database, and am using the
>>>>>> Dataimporthandler to add rows to the index...
>>>>>>
>>>>>> I am using solr 1.4
>>>>>>
>>>>>> I have added my the deleted field to the query and the
>>>>>> RegexTransformer...
>>>>>> and the field definition below
>>>>>>
>>>>>> <field column="$deleteDocByQuery"
>>>>>> regex="^true$"
>>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
>>>>>>
>>>>>> When I run the deltaImport command... I see the below output
>>>>>>
>>>>>> INFO: [] webapp=/solr path=/dataimport
>>>>>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
>>>>>> QTime=1
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.DataImporter
>>>>>> doDeltaImport
>>>>>> INFO: Starting Delta Import
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>>>>>> readIndexerProperties
>>>>>> INFO: Read dataimport.properties
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> doDelta
>>>>>> INFO: Starting delta collection.
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Running ModifiedRowKey() for Entity: List
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>>> call
>>>>>> INFO: Creating a connection for entity List with URL:
>>>>>> jdbc:postgresql://localhost:5432/tlists
>>>>>> Nov 16, 2009 5:29:10 PM
>>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>>> call
>>>>>> INFO: Time taken for getConnection(): 4
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> collectDelta
>>>>>> INFO: Completed parentDeltaQuery for Entity: List
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>>>>>> deleteByQuery
>>>>>> INFO: Deleting documents from Solr with query: id:api__list__365522
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
>>>>>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>>>>>> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
>>>>>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
>>>>>> _bg.frq]
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
>>>>>> updateCommits
>>>>>> INFO: newest commit = 1257863009839
>>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>>> doDelta
>>>>>> INFO: Delta Import completed successfully
>>>>>>
>>>>>> It says its deleting the document... but when I do the search its
>>>>>> still
>>>>>> showing up....
>>>>>>
>>>>>> Any Ideas?
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -----------------------------------------------------
>>>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> -----------------------------------------------------
>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
why don't you add a new timestamp field . you can use the
TemplateTransformer with the formatDate() function
On Tue, Nov 17, 2009 at 5:49 PM, Mark Ellul <ma...@catalystic.com> wrote:
> Hi Noble,
>
> Excellent Question... should the field that does the deleting be in a
> different entity to the one that does the addition and updating?
>
> If so that could be the issue, I have the field that does the
> DeleteByQuery command inside of the entity that does the adding.
>
> Is there some kind of document metadata where the create date and
> update date is show?
>
> How would I see this meta data if it exists?
>
> Regards
>
> Mark
>
> On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
>> The question is, did your new delta-import created the doc again?
>>
>> On Tue, Nov 17, 2009 at 4:41 PM, Mark Ellul <ma...@catalystic.com> wrote:
>>> The doc already existed before the delta-import has been run.
>>>
>>> And it exists afterwards... even though it says its deleting it.
>>>
>>> Any ideas of what I can try?
>>>
>>> On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
>>>> are you sure that the doc w/ the same id was not created after that?
>>>>
>>>> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul <ma...@catalystic.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I have added a deleted field in my database, and am using the
>>>>> Dataimporthandler to add rows to the index...
>>>>>
>>>>> I am using solr 1.4
>>>>>
>>>>> I have added my the deleted field to the query and the
>>>>> RegexTransformer...
>>>>> and the field definition below
>>>>>
>>>>> <field column="$deleteDocByQuery"
>>>>> regex="^true$"
>>>>> replaceWith="id:${List.id}" sourceColName="deleted"/>
>>>>>
>>>>> When I run the deltaImport command... I see the below output
>>>>>
>>>>> INFO: [] webapp=/solr path=/dataimport
>>>>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
>>>>> QTime=1
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
>>>>> doDeltaImport
>>>>> INFO: Starting Delta Import
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>>>>> readIndexerProperties
>>>>> INFO: Read dataimport.properties
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> doDelta
>>>>> INFO: Starting delta collection.
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> collectDelta
>>>>> INFO: Running ModifiedRowKey() for Entity: List
>>>>> Nov 16, 2009 5:29:10 PM
>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>> call
>>>>> INFO: Creating a connection for entity List with URL:
>>>>> jdbc:postgresql://localhost:5432/tlists
>>>>> Nov 16, 2009 5:29:10 PM
>>>>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>>>>> call
>>>>> INFO: Time taken for getConnection(): 4
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> collectDelta
>>>>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> collectDelta
>>>>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> collectDelta
>>>>> INFO: Completed parentDeltaQuery for Entity: List
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>>>>> deleteByQuery
>>>>> INFO: Deleting documents from Solr with query: id:api__list__365522
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
>>>>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>>>>> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
>>>>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis,
>>>>> _bg.frq]
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
>>>>> updateCommits
>>>>> INFO: newest commit = 1257863009839
>>>>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>>>>> doDelta
>>>>> INFO: Delta Import completed successfully
>>>>>
>>>>> It says its deleting the document... but when I do the search its still
>>>>> showing up....
>>>>>
>>>>> Any Ideas?
>>>>>
>>>>> Regards
>>>>>
>>>>> Mark
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -----------------------------------------------------
>>>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>>>
>>>
>>
>>
>>
>> --
>> -----------------------------------------------------
>> Noble Paul | Principal Engineer| AOL | http://aol.com
>>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Mark Ellul <ma...@catalystic.com>.
The doc already existed before the delta-import has been run.
And it exists afterwards... even though it says its deleting it.
Any ideas of what I can try?
On 11/17/09, Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com> wrote:
> are you sure that the doc w/ the same id was not created after that?
>
> On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul <ma...@catalystic.com> wrote:
>> Hi,
>>
>> I have added a deleted field in my database, and am using the
>> Dataimporthandler to add rows to the index...
>>
>> I am using solr 1.4
>>
>> I have added my the deleted field to the query and the RegexTransformer...
>> and the field definition below
>>
>> <field column="$deleteDocByQuery"
>> regex="^true$"
>> replaceWith="id:${List.id}" sourceColName="deleted"/>
>>
>> When I run the deltaImport command... I see the below output
>>
>> INFO: [] webapp=/solr path=/dataimport
>> params={command=delta-import&debug=true&expungeDeletes=true} status=0
>> QTime=1
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
>> doDeltaImport
>> INFO: Starting Delta Import
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>> readIndexerProperties
>> INFO: Read dataimport.properties
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> doDelta
>> INFO: Starting delta collection.
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Running ModifiedRowKey() for Entity: List
>> Nov 16, 2009 5:29:10 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Creating a connection for entity List with URL:
>> jdbc:postgresql://localhost:5432/tlists
>> Nov 16, 2009 5:29:10 PM
>> org.apache.solr.handler.dataimport.JdbcDataSource$1
>> call
>> INFO: Time taken for getConnection(): 4
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> collectDelta
>> INFO: Completed parentDeltaQuery for Entity: List
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
>> deleteByQuery
>> INFO: Deleting documents from Solr with query: id:api__list__365522
>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
>> INFO: SolrDeletionPolicy.onInit: commits:num=1
>> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
>> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
>> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
>> updateCommits
>> INFO: newest commit = 1257863009839
>> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
>> doDelta
>> INFO: Delta Import completed successfully
>>
>> It says its deleting the document... but when I do the search its still
>> showing up....
>>
>> Any Ideas?
>>
>> Regards
>>
>> Mark
>>
>
>
>
> --
> -----------------------------------------------------
> Noble Paul | Principal Engineer| AOL | http://aol.com
>
Re: $DeleteDocbyQuery in solr 1.4 is not working
Posted by Noble Paul നോബിള് नोब्ळ् <no...@corp.aol.com>.
are you sure that the doc w/ the same id was not created after that?
On Mon, Nov 16, 2009 at 11:12 PM, Mark Ellul <ma...@catalystic.com> wrote:
> Hi,
>
> I have added a deleted field in my database, and am using the
> Dataimporthandler to add rows to the index...
>
> I am using solr 1.4
>
> I have added my the deleted field to the query and the RegexTransformer...
> and the field definition below
>
> <field column="$deleteDocByQuery"
> regex="^true$"
> replaceWith="id:${List.id}" sourceColName="deleted"/>
>
> When I run the deltaImport command... I see the below output
>
> INFO: [] webapp=/solr path=/dataimport
> params={command=delta-import&debug=true&expungeDeletes=true} status=0
> QTime=1
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
> INFO: Starting Delta Import
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
> readIndexerProperties
> INFO: Read dataimport.properties
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> doDelta
> INFO: Starting delta collection.
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Running ModifiedRowKey() for Entity: List
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Creating a connection for entity List with URL:
> jdbc:postgresql://localhost:5432/tlists
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.JdbcDataSource$1
> call
> INFO: Time taken for getConnection(): 4
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed ModifiedRowKey for Entity: List rows obtained : 1
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed DeletedRowKey for Entity: List rows obtained : 0
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> collectDelta
> INFO: Completed parentDeltaQuery for Entity: List
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.SolrWriter
> deleteByQuery
> INFO: Deleting documents from Solr with query: id:api__list__365522
> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy onInit
> INFO: SolrDeletionPolicy.onInit: commits:num=1
> commit{dir=/mnt/solr-index/index,segFN=segments_r,version=1257863009839,generation=27,filenames=[_bg.fdt,
> _bg.tii, segments_r, _bg.fnm, _bg.nrm, _bg.fdx, _bg.prx, _bg.tis, _bg.frq]
> Nov 16, 2009 5:29:10 PM org.apache.solr.core.SolrDeletionPolicy
> updateCommits
> INFO: newest commit = 1257863009839
> Nov 16, 2009 5:29:10 PM org.apache.solr.handler.dataimport.DocBuilder
> doDelta
> INFO: Delta Import completed successfully
>
> It says its deleting the document... but when I do the search its still
> showing up....
>
> Any Ideas?
>
> Regards
>
> Mark
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com