You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Max Lynch <ih...@gmail.com> on 2010/08/26 00:12:35 UTC
Delete by query issue
Hi,
I am trying to delete all documents that have null values for a certain
field. To that effect I can see all of the documents I want to delete by
doing this query:
-date_added_solr:[* TO *]
This returns about 32,000 documents.
However, when I try to put that into a curl call, no documents get deleted:
curl http://localhost:8985/solr/newsblog/update?commit=true -H
"Content-Type: text/xml" --data-binary '<delete><query>-date_added_solr:[*
TO *]</query></delete>'
Solr responds with:
<response>
<lst name="responseHeader"><int name="status">0</int><int
name="QTime">364</int></lst>
</response>
But nothing happens, even if I explicitly issue a commit afterward.
Any ideas?
Thanks.
Re: Delete by query issue
Posted by Chris Hostetter <ho...@fucit.org>.
: Here's the problem: the standard Solr parser is a little weird about
: negative queries. The way to make this work is to say
: *:* AND -field:[* TO *]
the default parser actually works ok ... it's a bug specific to
deletion...
https://issues.apache.org/jira/browse/SOLR-381
-Hoss
--
http://lucenerevolution.org/ ... October 7-8, Boston
http://bit.ly/stump-hoss ... Stump The Chump!
Re: Delete by query issue
Posted by Max Lynch <ih...@gmail.com>.
Thanks Lance. I'll give that a try going forward.
On Wed, Aug 25, 2010 at 9:59 PM, Lance Norskog <go...@gmail.com> wrote:
> Here's the problem: the standard Solr parser is a little weird about
> negative queries. The way to make this work is to say
> *:* AND -field:[* TO *]
>
> This means "select everything AND only these documents without a value
> in the field".
>
> On Wed, Aug 25, 2010 at 7:55 PM, Max Lynch <ih...@gmail.com> wrote:
> > I was trying to filter out all documents that HAVE that field. I was
> trying
> > to delete any documents where that field had empty values.
> >
> > I just found a way to do it, but I did a range query on a string date in
> the
> > Lucene DateTools format and it worked, so I'm satisfied. However, I
> believe
> > it worked because all of my documents have values for that field.
> >
> > Oh well.
> >
> > -max
> >
> > On Wed, Aug 25, 2010 at 9:45 PM, scott chu (朱炎詹) <scott.chu@udngroup.com
> >wrote:
> >
> >> Excuse me, what's the hyphen before the field name 'date_added_solr'?
> Is
> >> this some kind of new query format that I didn't know?
> >>
> >> <delete><query>-date_added_solr:[* TO *]</query></delete>'
> >>
> >> ----- Original Message -----
> >> From: "Max Lynch" <ih...@gmail.com>
> >> To: <so...@lucene.apache.org>
> >> Sent: Thursday, August 26, 2010 6:12 AM
> >> Subject: Delete by query issue
> >>
> >>
> >> > Hi,
> >> > I am trying to delete all documents that have null values for a
> certain
> >> > field. To that effect I can see all of the documents I want to delete
> by
> >> > doing this query:
> >> > -date_added_solr:[* TO *]
> >> >
> >> > This returns about 32,000 documents.
> >> >
> >> > However, when I try to put that into a curl call, no documents get
> >> deleted:
> >> > curl http://localhost:8985/solr/newsblog/update?commit=true -H
> >> > "Content-Type: text/xml" --data-binary
> >> '<delete><query>-date_added_solr:[*
> >> > TO *]</query></delete>'
> >> >
> >> > Solr responds with:
> >> > <response>
> >> > <lst name="responseHeader"><int name="status">0</int><int
> >> > name="QTime">364</int></lst>
> >> > </response>
> >> >
> >> > But nothing happens, even if I explicitly issue a commit afterward.
> >> >
> >> > Any ideas?
> >> >
> >> > Thanks.
> >> >
> >>
> >>
> >>
> >>
> --------------------------------------------------------------------------------
> >>
> >>
> >>
> >> %<&b6G$J0T.'$$'d(l/f,r!C
> >> Checked by AVG - www.avg.com
> >> Version: 9.0.851 / Virus Database: 271.1.1/3093 - Release Date: 08/25/10
> >> 14:34:00
> >>
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
Re: Delete by query issue
Posted by Lance Norskog <go...@gmail.com>.
Here's the problem: the standard Solr parser is a little weird about
negative queries. The way to make this work is to say
*:* AND -field:[* TO *]
This means "select everything AND only these documents without a value
in the field".
On Wed, Aug 25, 2010 at 7:55 PM, Max Lynch <ih...@gmail.com> wrote:
> I was trying to filter out all documents that HAVE that field. I was trying
> to delete any documents where that field had empty values.
>
> I just found a way to do it, but I did a range query on a string date in the
> Lucene DateTools format and it worked, so I'm satisfied. However, I believe
> it worked because all of my documents have values for that field.
>
> Oh well.
>
> -max
>
> On Wed, Aug 25, 2010 at 9:45 PM, scott chu (朱炎詹) <sc...@udngroup.com>wrote:
>
>> Excuse me, what's the hyphen before the field name 'date_added_solr'? Is
>> this some kind of new query format that I didn't know?
>>
>> <delete><query>-date_added_solr:[* TO *]</query></delete>'
>>
>> ----- Original Message -----
>> From: "Max Lynch" <ih...@gmail.com>
>> To: <so...@lucene.apache.org>
>> Sent: Thursday, August 26, 2010 6:12 AM
>> Subject: Delete by query issue
>>
>>
>> > Hi,
>> > I am trying to delete all documents that have null values for a certain
>> > field. To that effect I can see all of the documents I want to delete by
>> > doing this query:
>> > -date_added_solr:[* TO *]
>> >
>> > This returns about 32,000 documents.
>> >
>> > However, when I try to put that into a curl call, no documents get
>> deleted:
>> > curl http://localhost:8985/solr/newsblog/update?commit=true -H
>> > "Content-Type: text/xml" --data-binary
>> '<delete><query>-date_added_solr:[*
>> > TO *]</query></delete>'
>> >
>> > Solr responds with:
>> > <response>
>> > <lst name="responseHeader"><int name="status">0</int><int
>> > name="QTime">364</int></lst>
>> > </response>
>> >
>> > But nothing happens, even if I explicitly issue a commit afterward.
>> >
>> > Any ideas?
>> >
>> > Thanks.
>> >
>>
>>
>>
>> --------------------------------------------------------------------------------
>>
>>
>>
>> %<&b6G$J0T.'$$'d(l/f,r!C
>> Checked by AVG - www.avg.com
>> Version: 9.0.851 / Virus Database: 271.1.1/3093 - Release Date: 08/25/10
>> 14:34:00
>>
>
--
Lance Norskog
goksron@gmail.com
Re: Delete by query issue
Posted by Max Lynch <ih...@gmail.com>.
I was trying to filter out all documents that HAVE that field. I was trying
to delete any documents where that field had empty values.
I just found a way to do it, but I did a range query on a string date in the
Lucene DateTools format and it worked, so I'm satisfied. However, I believe
it worked because all of my documents have values for that field.
Oh well.
-max
On Wed, Aug 25, 2010 at 9:45 PM, scott chu (朱炎詹) <sc...@udngroup.com>wrote:
> Excuse me, what's the hyphen before the field name 'date_added_solr'? Is
> this some kind of new query format that I didn't know?
>
> <delete><query>-date_added_solr:[* TO *]</query></delete>'
>
> ----- Original Message -----
> From: "Max Lynch" <ih...@gmail.com>
> To: <so...@lucene.apache.org>
> Sent: Thursday, August 26, 2010 6:12 AM
> Subject: Delete by query issue
>
>
> > Hi,
> > I am trying to delete all documents that have null values for a certain
> > field. To that effect I can see all of the documents I want to delete by
> > doing this query:
> > -date_added_solr:[* TO *]
> >
> > This returns about 32,000 documents.
> >
> > However, when I try to put that into a curl call, no documents get
> deleted:
> > curl http://localhost:8985/solr/newsblog/update?commit=true -H
> > "Content-Type: text/xml" --data-binary
> '<delete><query>-date_added_solr:[*
> > TO *]</query></delete>'
> >
> > Solr responds with:
> > <response>
> > <lst name="responseHeader"><int name="status">0</int><int
> > name="QTime">364</int></lst>
> > </response>
> >
> > But nothing happens, even if I explicitly issue a commit afterward.
> >
> > Any ideas?
> >
> > Thanks.
> >
>
>
>
> --------------------------------------------------------------------------------
>
>
>
> %<&b6G$J0T.'$$'d(l/f,r!C
> Checked by AVG - www.avg.com
> Version: 9.0.851 / Virus Database: 271.1.1/3093 - Release Date: 08/25/10
> 14:34:00
>
Re: Is there any strss test tool for testing Solr?
Posted by Chris Hostetter <ho...@fucit.org>.
: References: <AA...@mail.gmail.com>
: <C7...@udc70634P002>
: In-Reply-To: <C7...@udc70634P002>
: Subject: Is there any strss test tool for testing Solr?
http://people.apache.org/~hossman/#threadhijack
Thread Hijacking on Mailing Lists
When starting a new discussion on a mailing list, please do not reply to
an existing message, instead start a fresh email. Even if you change the
subject line of your email, other mail headers still track which thread
you replied to and your question is "hidden" in that thread and gets less
attention. It makes following discussions in the mailing list archives
particularly difficult.
See Also: http://en.wikipedia.org/wiki/User:DonDiego/Thread_hijacking
-Hoss
--
http://lucenerevolution.org/ ... October 7-8, Boston
http://bit.ly/stump-hoss ... Stump The Chump!
Re: Is there any strss test tool for testing Solr?
Posted by "scott chu (朱炎詹)" <sc...@udngroup.com>.
Thanks to both Gora & Amit. A little information for people who concern this
discussion, I found there's a SolrMeter open source project in Google Code -
http://code.google.com/p/solrmeter/, it's specifically for load test of
Solr -
I'll evaluate following tools & pick up one for my testing:
WebStress
Apache Bench
JMeter
SolrMetre
Oh, I'll correct a wrong information my post: We're builiding a 12-million
newspaper index, rather than 1.2 million.
Scott
----- Original Message -----
From: "Gora Mohanty" <go...@srijan.in>
To: <so...@lucene.apache.org>
Sent: Friday, August 27, 2010 2:22 AM
Subject: Re: Is there any strss test tool for testing Solr?
> On Wed, 25 Aug 2010 19:58:36 -0700
> Amit Nithian <an...@gmail.com> wrote:
>
>> i recommend JMeter. We use that to do load testing on a search
>> server.
> [...]
>
> JMeter is certainly good, but we have also found Apache bench
> to also be of much use. Maybe it is just us, and what we are
> familiar with, but Apache bench seemed easier to automate. Also,
> much easier to get up and running with, at least IMHO.
>
>> Be careful though.. as silly as this may sound.. do NOT just
>> issue random queries because that won't exercise your caches...
> [...]
>
> Conversely, we are still trying to figure out how to make real-life
> measurements, without having the Solr cache coming into the picture.
> For querying on a known keyword, every hit after the first, with
> Apache bench, is strongly affected by the Solr cache. We tried using
> random strings, but at least with Apache bench, the query string is
> fixed for each invocation of Apache bench. Have to investigate
> whether one can do otherwise with JMeter plugins. Also, a query
> that returns no result (as a random query string typically would)
> seems to be significantly faster than a real query. So, I think that
> in the long run, the best way is to build information about
> *typical* queries that your users run; using the Solr logs, and
> then use a set of such queries for benchmarking.
>
> Regards,
> Gora
>
--------------------------------------------------------------------------------
¥¼¦b¶Ç¤J°T®§¤¤§ä¨ì¯f¬r¡C
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3094 - Release Date: 08/26/10
02:34:00
Re: Is there any strss test tool for testing Solr?
Posted by Gora Mohanty <go...@srijan.in>.
On Wed, 25 Aug 2010 19:58:36 -0700
Amit Nithian <an...@gmail.com> wrote:
> i recommend JMeter. We use that to do load testing on a search
> server.
[...]
JMeter is certainly good, but we have also found Apache bench
to also be of much use. Maybe it is just us, and what we are
familiar with, but Apache bench seemed easier to automate. Also,
much easier to get up and running with, at least IMHO.
> Be careful though.. as silly as this may sound.. do NOT just
> issue random queries because that won't exercise your caches...
[...]
Conversely, we are still trying to figure out how to make real-life
measurements, without having the Solr cache coming into the picture.
For querying on a known keyword, every hit after the first, with
Apache bench, is strongly affected by the Solr cache. We tried using
random strings, but at least with Apache bench, the query string is
fixed for each invocation of Apache bench. Have to investigate
whether one can do otherwise with JMeter plugins. Also, a query
that returns no result (as a random query string typically would)
seems to be significantly faster than a real query. So, I think that
in the long run, the best way is to build information about
*typical* queries that your users run; using the Solr logs, and
then use a set of such queries for benchmarking.
Regards,
Gora
Re: Is there any strss test tool for testing Solr?
Posted by Amit Nithian <an...@gmail.com>.
i recommend JMeter. We use that to do load testing on a search server. of
course you have to provide a reasonable set of queries as input... if you
don't have any then a reasonable estimation based on your expected traffic
should suffice. JMeter can be used for other load testing too..
Be careful though.. as silly as this may sound.. do NOT just issue random
queries because that won't exercise your caches... We had a load test that
killed our servers because our caches kept getting blown out. Of course the
traffic being generated was purely random was not representative of real
world traffic which usually has more predictable behavior.
hope that helps!
Amit
On Wed, Aug 25, 2010 at 7:50 PM, scott chu (朱炎詹) <sc...@udngroup.com>wrote:
> We're currently building a Solr index with ober 1.2 million documents. I
> want to do a good stress test of it. Does anyone know if ther's a
> appropriate stress test tool for Solr? Or any good suggestion?
>
> Best Regards,
>
> Scott
>
Is there any strss test tool for testing Solr?
Posted by "scott chu (朱炎詹)" <sc...@udngroup.com>.
We're currently building a Solr index with ober 1.2 million documents. I
want to do a good stress test of it. Does anyone know if ther's a
appropriate stress test tool for Solr? Or any good suggestion?
Best Regards,
Scott
Re: Delete by query issue
Posted by "scott chu (朱炎詹)" <sc...@udngroup.com>.
Excuse me, what's the hyphen before the field name 'date_added_solr'? Is this some kind of new query format that I didn't know?
<delete><query>-date_added_solr:[* TO *]</query></delete>'
----- Original Message -----
From: "Max Lynch" <ih...@gmail.com>
To: <so...@lucene.apache.org>
Sent: Thursday, August 26, 2010 6:12 AM
Subject: Delete by query issue
> Hi,
> I am trying to delete all documents that have null values for a certain
> field. To that effect I can see all of the documents I want to delete by
> doing this query:
> -date_added_solr:[* TO *]
>
> This returns about 32,000 documents.
>
> However, when I try to put that into a curl call, no documents get deleted:
> curl http://localhost:8985/solr/newsblog/update?commit=true -H
> "Content-Type: text/xml" --data-binary '<delete><query>-date_added_solr:[*
> TO *]</query></delete>'
>
> Solr responds with:
> <response>
> <lst name="responseHeader"><int name="status">0</int><int
> name="QTime">364</int></lst>
> </response>
>
> But nothing happens, even if I explicitly issue a commit afterward.
>
> Any ideas?
>
> Thanks.
>
--------------------------------------------------------------------------------
%<&b6G$J0T.'$$'d(l/f,r!C
Checked by AVG - www.avg.com
Version: 9.0.851 / Virus Database: 271.1.1/3093 - Release Date: 08/25/10 14:34:00