You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Frederico Azeiteiro <Fr...@cision.com> on 2010/02/03 12:33:47 UTC
query all filled field?
Hi all,
Is it possible to query some field in order to get only not empty
documents?
All documents where field x is filled?
Thanks,
Frederico
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
Ok, if anyone needs it:
I tried fieldX:[* TO *]
I think this is correct.
In my case I found out that I was not indexing this field correctly
because they are all empty. :)
-----Original Message-----
From: Frederico Azeiteiro [mailto:Frederico.Azeiteiro@cision.com]
Sent: quarta-feira, 3 de Fevereiro de 2010 11:34
To: solr-user@lucene.apache.org
Subject: query all filled field?
Hi all,
Is it possible to query some field in order to get only not empty
documents?
All documents where field x is filled?
Thanks,
Frederico
RE: query all filled field?
Posted by Ahmet Arslan <io...@yahoo.com>.
> I've analyzed my index application
> and checked the XML before executing the http request and
> the field it's empty:
>
> <field name="fieldX" />
>
> It should be empty on SOLR.
>
> Probably something in the way between my application (.NET)
> and the SOLR (Jetty on Ubuntu) adds the whitespace.
>
> Anyway, I'll try to remove the field but, as I validate
> each doc to the SOLR schema, I must make some adjustments
> and stop validate the doc.
> I don't know if that will be acceptable...
>
> Thanks for your help.
Alternatively you can do this in solr side with an UpdateProcessor [1], something like:
public void processAdd(AddUpdateCommand cmd) throws IOException {
SolrInputDocument doc = cmd.getSolrInputDocument();
Object v = doc.getFieldValue( "fieldX" );
if( v != null ) {
String fieldX = (String) v.toString();
if("".equals(fieldX.trim()) ) {
doc.removeField( "fieldX" );
}
}
super.processAdd(cmd);
}
[1] http://wiki.apache.org/solr/UpdateRequestProcessor
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
I've analyzed my index application and checked the XML before executing the http request and the field it's empty:
<field name="fieldX" />
It should be empty on SOLR.
Probably something in the way between my application (.NET) and the SOLR (Jetty on Ubuntu) adds the whitespace.
Anyway, I'll try to remove the field but, as I validate each doc to the SOLR schema, I must make some adjustments and stop validate the doc.
I don't know if that will be acceptable...
Thanks for your help.
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 17:22
To: solr-user@lucene.apache.org
Subject: RE: query all filled field?
> XML update. I'm serializing the doc
> in .NET, and then using solsharp to
> insert/update the doc to SOLR.
>
> The result is:
>
> <doc>
> <str name="fieldX"/>
> </doc>
>
> Dows this means I'm adding a whitespace on XML Update?
Yes exactly. You can remove <field name="fieldX"> </field> from your
<add>
<doc>
...
</doc>
</add>
if value of fieldX.trim() is equal to "" when preparing your xml.
RE: query all filled field?
Posted by Ahmet Arslan <io...@yahoo.com>.
> XML update. I'm serializing the doc
> in .NET, and then using solsharp to
> insert/update the doc to SOLR.
>
> The result is:
>
> <doc>
> <str name="fieldX"/>
> </doc>
>
> Dows this means I'm adding a whitespace on XML Update?
Yes exactly. You can remove <field name="fieldX"> </field> from your
<add>
<doc>
...
</doc>
</add>
if value of fieldX.trim() is equal to "" when preparing your xml.
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
XML update. I'm serializing the doc in .NET, and then using solsharp to
insert/update the doc to SOLR.
The result is:
<doc>
<str name="fieldX"/>
</doc>
Dows this means I'm adding a whitespace on XML Update?
Frederico
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 16:26
To: solr-user@lucene.apache.org
Subject: RE: query all filled field?
> Theoretically yes,it's correct, but i
> have about 1/10 of the docs with
> this field not empty and the rest is empty.
>
> Most of the articles have the field empty as I can see when
> query *:*.
How are you adding documents to solr? xml update, DIH?
Probably you are adding whitespace value to that field.
When you query q=*:*&fl=fieldX what do you see?
Do you see
<doc>
<str name="fieldX"></str>
</doc>
or
<doc>
</doc>
RE: query all filled field?
Posted by Ahmet Arslan <io...@yahoo.com>.
> Theoretically yes,it's correct, but i
> have about 1/10 of the docs with
> this field not empty and the rest is empty.
>
> Most of the articles have the field empty as I can see when
> query *:*.
How are you adding documents to solr? xml update, DIH?
Probably you are adding whitespace value to that field.
When you query q=*:*&fl=fieldX what do you see?
Do you see
<doc>
<str name="fieldX"></str>
</doc>
or
<doc>
</doc>
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
Theoretically yes,it's correct, but i have about 1/10 of the docs with
this field not empty and the rest is empty.
Most of the articles have the field empty as I can see when query *:*.
So the queries don't make sense...
-----Original Message-----
From: Ankit Bhatnagar [mailto:abhatnagar@vantage.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 14:56
To: 'solr-user@lucene.apache.org'
Subject: RE: query all filled field?
That's correct.
If u want to find "Missing Values"
ie fields for whom value is not present then u will use -
Ankit
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: Thursday, February 04, 2010 9:41 AM
To: solr-user@lucene.apache.org
Subject: RE: query all filled field?
> *:* AND -fieldX:[* TO *] - returns 0 docs
>
> fieldX:(a*) - return docs, so I'm sure that there's docs
> with this field filled.
>
> Any other ideias what could be wrong?
There is not wrong in this scenario.
If -fieldX:[* TO *] returns 0 docs, it means that all of your documents
have that fieldX filled. Therefore fieldX:[* TO *] returns all of your
dataset same as *:*
RE: query all filled field?
Posted by Ankit Bhatnagar <ab...@vantage.com>.
That's correct.
If u want to find "Missing Values"
ie fields for whom value is not present then u will use -
Ankit
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: Thursday, February 04, 2010 9:41 AM
To: solr-user@lucene.apache.org
Subject: RE: query all filled field?
> *:* AND -fieldX:[* TO *] - returns 0 docs
>
> fieldX:(a*) - return docs, so I'm sure that there's docs
> with this field filled.
>
> Any other ideias what could be wrong?
There is not wrong in this scenario.
If -fieldX:[* TO *] returns 0 docs, it means that all of your documents have that fieldX filled. Therefore fieldX:[* TO *] returns all of your dataset same as *:*
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
I tried another one:
fieldX:["" TO *] and it returns articles with the field filled :), so I guess I'm getting there.
But I tried also fieldX:[" " TO *] and get a few more results that the first one...
Is there a real difference between these, and also if the results are really all docs with field not empty?
Thanks again,
Frederico
-----Original Message-----
From: Frederico Azeiteiro [mailto:Frederico.Azeiteiro@cision.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 10:55
To: solr-user@lucene.apache.org
Subject: RE: query all filled field?
Thanks, but still no luck with that:
*:* AND -fieldX:[* TO *] - returns 0 docs
fieldX:(a*) - return docs, so I'm sure that there's docs with this field filled.
Any other ideias what could be wrong?
Frederico
-----Original Message-----
From: Lance Norskog [mailto:goksron@gmail.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 05:38
To: solr-user@lucene.apache.org
Subject: Re: query all filled field?
Queries that start with minus or NOT don't work. You have to do this:
*:* AND -fieldX:[* TO *]
On Wed, Feb 3, 2010 at 5:04 AM, Frederico Azeiteiro
<Fr...@cision.com> wrote:
> Hum, strange.. I reindexed some docs with the field corrected.
>
> Now I'm sure the field is filled because:
>
> "fieldX:(*a*)" returns docs.
>
> But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
>
> I tried with "-fieldX:[* TO *]" and I get no results at all.
>
> I wonder if someone has tried this before with success?
>
> The field is indexed as string, indexed=true and stored=true.
>
> Thanks,
> Frederico
>
> -----Original Message-----
> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
> Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
> To: solr-user@lucene.apache.org
> Subject: Re: query all filled field?
>
>
>> Is it possible to query some field in order to get only not
>> empty
>> documents?
>>
>>
>>
>> All documents where field x is filled?
>
> Yes. q=x:[* TO *] will bring documents that has non-empty x field.
>
>
>
>
--
Lance Norskog
goksron@gmail.com
RE: query all filled field?
Posted by Ahmet Arslan <io...@yahoo.com>.
> *:* AND -fieldX:[* TO *] - returns 0 docs
>
> fieldX:(a*) - return docs, so I'm sure that there's docs
> with this field filled.
>
> Any other ideias what could be wrong?
There is not wrong in this scenario.
If -fieldX:[* TO *] returns 0 docs, it means that all of your documents have that fieldX filled. Therefore fieldX:[* TO *] returns all of your dataset same as *:*
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
Thanks, but still no luck with that:
*:* AND -fieldX:[* TO *] - returns 0 docs
fieldX:(a*) - return docs, so I'm sure that there's docs with this field filled.
Any other ideias what could be wrong?
Frederico
-----Original Message-----
From: Lance Norskog [mailto:goksron@gmail.com]
Sent: quinta-feira, 4 de Fevereiro de 2010 05:38
To: solr-user@lucene.apache.org
Subject: Re: query all filled field?
Queries that start with minus or NOT don't work. You have to do this:
*:* AND -fieldX:[* TO *]
On Wed, Feb 3, 2010 at 5:04 AM, Frederico Azeiteiro
<Fr...@cision.com> wrote:
> Hum, strange.. I reindexed some docs with the field corrected.
>
> Now I'm sure the field is filled because:
>
> "fieldX:(*a*)" returns docs.
>
> But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
>
> I tried with "-fieldX:[* TO *]" and I get no results at all.
>
> I wonder if someone has tried this before with success?
>
> The field is indexed as string, indexed=true and stored=true.
>
> Thanks,
> Frederico
>
> -----Original Message-----
> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
> Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
> To: solr-user@lucene.apache.org
> Subject: Re: query all filled field?
>
>
>> Is it possible to query some field in order to get only not
>> empty
>> documents?
>>
>>
>>
>> All documents where field x is filled?
>
> Yes. q=x:[* TO *] will bring documents that has non-empty x field.
>
>
>
>
--
Lance Norskog
goksron@gmail.com
Re: query all filled field?
Posted by Erik Hatcher <er...@gmail.com>.
On Feb 4, 2010, at 12:38 AM, Lance Norskog wrote:
> Queries that start with minus or NOT don't work. You have to do this:
> *:* AND -fieldX:[* TO *]
That's only true for subqueries. A purely negative single top-level
clause works fine with Solr.
Erik
>
> On Wed, Feb 3, 2010 at 5:04 AM, Frederico Azeiteiro
> <Fr...@cision.com> wrote:
>> Hum, strange.. I reindexed some docs with the field corrected.
>>
>> Now I'm sure the field is filled because:
>>
>> "fieldX:(*a*)" returns docs.
>>
>> But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
>>
>> I tried with "-fieldX:[* TO *]" and I get no results at all.
>>
>> I wonder if someone has tried this before with success?
>>
>> The field is indexed as string, indexed=true and stored=true.
>>
>> Thanks,
>> Frederico
>>
>> -----Original Message-----
>> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
>> Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
>> To: solr-user@lucene.apache.org
>> Subject: Re: query all filled field?
>>
>>
>>> Is it possible to query some field in order to get only not
>>> empty
>>> documents?
>>>
>>>
>>>
>>> All documents where field x is filled?
>>
>> Yes. q=x:[* TO *] will bring documents that has non-empty x field.
>>
>>
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: query all filled field?
Posted by Mark Miller <ma...@gmail.com>.
Not entirely true - thats the case in Lucene, but in Solr, top level
queries *can* start with minus or not. They cannot if they are nested.
Both
*:* AND -fieldX:[* TO *]
and
-fieldX:[* TO *]
are the same in Solr.
--
- Mark
http://www.lucidimagination.com
Lance Norskog wrote:
> Queries that start with minus or NOT don't work. You have to do this:
> *:* AND -fieldX:[* TO *]
>
> On Wed, Feb 3, 2010 at 5:04 AM, Frederico Azeiteiro
> <Fr...@cision.com> wrote:
>
>> Hum, strange.. I reindexed some docs with the field corrected.
>>
>> Now I'm sure the field is filled because:
>>
>> "fieldX:(*a*)" returns docs.
>>
>> But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
>>
>> I tried with "-fieldX:[* TO *]" and I get no results at all.
>>
>> I wonder if someone has tried this before with success?
>>
>> The field is indexed as string, indexed=true and stored=true.
>>
>> Thanks,
>> Frederico
>>
>> -----Original Message-----
>> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
>> Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
>> To: solr-user@lucene.apache.org
>> Subject: Re: query all filled field?
>>
>>
>>
>>> Is it possible to query some field in order to get only not
>>> empty
>>> documents?
>>>
>>>
>>>
>>> All documents where field x is filled?
>>>
>> Yes. q=x:[* TO *] will bring documents that has non-empty x field.
>>
>>
>>
>>
>>
>
>
>
>
Re: query all filled field?
Posted by Lance Norskog <go...@gmail.com>.
Queries that start with minus or NOT don't work. You have to do this:
*:* AND -fieldX:[* TO *]
On Wed, Feb 3, 2010 at 5:04 AM, Frederico Azeiteiro
<Fr...@cision.com> wrote:
> Hum, strange.. I reindexed some docs with the field corrected.
>
> Now I'm sure the field is filled because:
>
> "fieldX:(*a*)" returns docs.
>
> But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
>
> I tried with "-fieldX:[* TO *]" and I get no results at all.
>
> I wonder if someone has tried this before with success?
>
> The field is indexed as string, indexed=true and stored=true.
>
> Thanks,
> Frederico
>
> -----Original Message-----
> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
> Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
> To: solr-user@lucene.apache.org
> Subject: Re: query all filled field?
>
>
>> Is it possible to query some field in order to get only not
>> empty
>> documents?
>>
>>
>>
>> All documents where field x is filled?
>
> Yes. q=x:[* TO *] will bring documents that has non-empty x field.
>
>
>
>
--
Lance Norskog
goksron@gmail.com
RE: query all filled field?
Posted by Frederico Azeiteiro <Fr...@cision.com>.
Hum, strange.. I reindexed some docs with the field corrected.
Now I'm sure the field is filled because:
"fieldX:(*a*)" returns docs.
But "fieldX:[* TO *]" is returning the same as "*.*" (all results)
I tried with "-fieldX:[* TO *]" and I get no results at all.
I wonder if someone has tried this before with success?
The field is indexed as string, indexed=true and stored=true.
Thanks,
Frederico
-----Original Message-----
From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
Sent: quarta-feira, 3 de Fevereiro de 2010 11:48
To: solr-user@lucene.apache.org
Subject: Re: query all filled field?
> Is it possible to query some field in order to get only not
> empty
> documents?
>
>
>
> All documents where field x is filled?
Yes. q=x:[* TO *] will bring documents that has non-empty x field.
Re: query all filled field?
Posted by Ahmet Arslan <io...@yahoo.com>.
> Is it possible to query some field in order to get only not
> empty
> documents?
>
>
>
> All documents where field x is filled?
Yes. q=x:[* TO *] will bring documents that has non-empty x field.