You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Marcus Stratmann <st...@gmx.de> on 2006/04/11 20:26:43 UTC
Deleting documents
Hello,
I have a problem deleting documents from the index.
In the tutorial "<delete><id>SP2514N</id></delete>" is used as an
example for deleting. I was wondering if "<id>" is some kind of
keyword or the name of a field (in the example, a unique field
named "id" is used). In my config I have the line
<uniqueKey>bookID</uniqueKey>
making bookID (type slong as defined in the example config) my
unique id.
So I tried
<delete><bookID>113976235</bookID></delete>
which resulted in
<result status="400">unexpected XML tag /delete/bookID</result>
Okay, so "id" seems to be a keyword, rather than a field name.
With my next try
<delete><id>113976235</id></delete>
the query worked fine:
<result status="0"></result>
But after a <commit/> I found the number of documents unchanged
in the stats. Furthermore the value of deletesById was 0.
Oddly enough cumulative_deletesById was 1 (what does this value
actually mean?).
Any ideas what's going wrong?
Thanks,
Marcus
Re: Deleting documents
Posted by Yonik Seeley <ys...@gmail.com>.
On 4/12/06, Marcus Stratmann <st...@gmx.de> wrote:
> > Yes, I believe the Wiki has an example like this (a uniqueKey field
> > not named "id")
> Right, I should have looked there, too.
>
> > > But after a <commit/> I found the number of documents unchanged
> > > in the stats.
If numDocs didn't change, nothing was deleted (this number is obtained
directly from Lucene).
Try verifying the id actually exists by doing a query on it.
Then after you delete and commit, verify that the id is gone by
re-executing that query.
-Yonik
Re: Deleting documents
Posted by Chris Hostetter <ho...@fucit.org>.
: OK, I think I fixed this bug. Haven't added a test case yet...
: I'll get to that sometime unless someone beats me to it.
: It does sort of require a separate schema to test though (which out
: test harness doesn't really handle yet, I think)
actually the test harness *can* handle differnet schema.xml files for
different test classes (just not differnet solrconfig.xml files)
but i haven't moved/copied src/apps/SolrTest/solr/conf/schema.xml to the
test directory becausei was waiting to solve the solrconfig.xml problem
first.
: On 4/12/06, Yonik Seeley <ys...@gmail.com> wrote:
: > On 4/12/06, Chris Hostetter <ho...@fucit.org> wrote:
: > >
: > > : Could it be the problem is that my unique key field
: > > : is of type slong (as defined in the tutorial)?
: > >
: > > I just tried modifying the example schema to use type slong for the "id"
: > > field, and i can reproduce what you are describing -- numDocs doesn't go
: > > down, ... because the delete isn't working.
: > >
: > > I'm guessing the problem is that the code for deleting by id is assuming
: > > it can do a delete b term (which it can) but isn't running the string
: > > through the FieldType to get the internalized value first.
: >
: > Huh.. right you are. I guess we've only used strings for ids before.
: > Marcus, unless you need to do range queries on your id field for some
: > reason, I'd suggest changing it to a string type.
: >
: > In the meantime, I'll fix that bug.
: >
: > -Yonik
:
-Hoss
Re: Deleting documents
Posted by Marcus Stratmann <st...@gmx.de>.
Yonik Seeley wrote:
> OK, I think I fixed this bug. Haven't added a test case yet...
In our test case everything works properly now.
Thanks for the quick bugfix!
Marcus
Re: Deleting documents
Posted by Yonik Seeley <ys...@gmail.com>.
OK, I think I fixed this bug. Haven't added a test case yet...
I'll get to that sometime unless someone beats me to it.
It does sort of require a separate schema to test though (which out
test harness doesn't really handle yet, I think)
-Yonik
On 4/12/06, Yonik Seeley <ys...@gmail.com> wrote:
> On 4/12/06, Chris Hostetter <ho...@fucit.org> wrote:
> >
> > : Could it be the problem is that my unique key field
> > : is of type slong (as defined in the tutorial)?
> >
> > I just tried modifying the example schema to use type slong for the "id"
> > field, and i can reproduce what you are describing -- numDocs doesn't go
> > down, ... because the delete isn't working.
> >
> > I'm guessing the problem is that the code for deleting by id is assuming
> > it can do a delete b term (which it can) but isn't running the string
> > through the FieldType to get the internalized value first.
>
> Huh.. right you are. I guess we've only used strings for ids before.
> Marcus, unless you need to do range queries on your id field for some
> reason, I'd suggest changing it to a string type.
>
> In the meantime, I'll fix that bug.
>
> -Yonik
Re: Deleting documents
Posted by Yonik Seeley <ys...@gmail.com>.
On 4/12/06, Chris Hostetter <ho...@fucit.org> wrote:
>
> : Could it be the problem is that my unique key field
> : is of type slong (as defined in the tutorial)?
>
> I just tried modifying the example schema to use type slong for the "id"
> field, and i can reproduce what you are describing -- numDocs doesn't go
> down, ... because the delete isn't working.
>
> I'm guessing the problem is that the code for deleting by id is assuming
> it can do a delete b term (which it can) but isn't running the string
> through the FieldType to get the internalized value first.
Huh.. right you are. I guess we've only used strings for ids before.
Marcus, unless you need to do range queries on your id field for some
reason, I'd suggest changing it to a string type.
In the meantime, I'll fix that bug.
-Yonik
Re: Deleting documents
Posted by Chris Hostetter <ho...@fucit.org>.
: Could it be the problem is that my unique key field
: is of type slong (as defined in the tutorial)?
I just tried modifying the example schema to use type slong for the "id"
field, and i can reproduce what you are describing -- numDocs doesn't go
down, ... because the delete isn't working.
I'm guessing the problem is that the code for deleting by id is assuming
it can do a delete b term (which it can) but isn't running the string
through the FieldType to get the internalized value first.
Can you file a bug for this in Jira so we can track it properly?
-Hoss
Re: Deleting documents
Posted by Marcus Stratmann <st...@gmx.de>.
> Yes, I believe the Wiki has an example like this (a uniqueKey field
> not named "id")
Right, I should have looked there, too.
> > But after a <commit/> I found the number of documents unchanged
> > in the stats.
> What stat? maxDoc may be unchanged since it doesn't reflect deleted
> documents that haven't been squeezed out of the index (it's a lucene
> thing). numDocs should reflect the deletion.
Yep, but numDocs is unchanged after a commit.
I tried it again this morning step by step.
Starting with a newly created index, the stats say
numDocs : 9882062
maxDoc : 9882062
commits : 0
optimizes : 0
docsPending : 0
deletesPending : 0
adds : 0
deletesById : 0
deletesByQuery : 0
errors : 0
cumulative_adds : 0
cumulative_deletesById : 0
cumulative_deletesByQuery : 0
cumulative_errors : 0
docsDeleted : 0
After giving the delete-command this changed to
numDocs : 9882062
maxDoc : 9882062
commits : 0
optimizes : 0
docsPending : 0
deletesPending : 1
adds : 0
deletesById : 1
deletesByQuery : 0
errors : 0
cumulative_adds : 0
cumulative_deletesById : 1
cumulative_deletesByQuery : 0
cumulative_errors : 0
docsDeleted : 0
And yes, I am absolutely sure the id I used for deletion
existed in the index. I tried it later with a delete by
query and it worked. (I just mention this because I found
out that the stats look like that regardless of which id
you use, an existing or non-existing one.)
Finally after a commit I got:
numDocs : 9882062
maxDoc : 9882062
commits : 1
optimizes : 0
docsPending : 0
deletesPending : 0
adds : 0
deletesById : 0
deletesByQuery : 0
errors : 0
cumulative_adds : 0
cumulative_deletesById : 1
cumulative_deletesByQuery : 0
cumulative_errors : 0
docsDeleted : 0
Apparently no change in the number of documents (and
the document can still be found in the index).
Could it be the problem is that my unique key field
is of type slong (as defined in the tutorial)?
Thanks,
Marcus
--
GMX Produkte empfehlen und ganz einfach Geld verdienen!
Satte Provisionen f�r GMX Partner: http://www.gmx.net/de/go/partner
Re: Deleting documents
Posted by Yonik Seeley <ys...@gmail.com>.
On 4/11/06, Marcus Stratmann <st...@gmx.de> wrote:
> Okay, so "id" seems to be a keyword, rather than a field name.
Yes, I believe the Wiki has an example like this (a uniqueKey field
not named "id")
> With my next try
> <delete><id>113976235</id></delete>
> the query worked fine:
> <result status="0"></result>
> But after a <commit/> I found the number of documents unchanged
> in the stats.
What stat? maxDoc may be unchanged since it doesn't reflect deleted
documents that haven't been squeezed out of the index (it's a lucene
thing). numDocs should reflect the deletion.
> Furthermore the value of deletesById was 0.
> Oddly enough cumulative_deletesById was 1 (what does this value
> actually mean?).
If you had looked at the stats before doing a <commit/>, deletesById
would probably be 1.
It gets reset on a <commit/> cumulative stats are never reset.
-Yonik