facet.missing=true returns null records with zero count also

[Rahul R <ra...@gmail.com>]

All,
We had a requirement in our solr powered application where customers want
to see all the documents that have a blank value for a field. So when they
facet on a field, if the field has null values, they should be able select
that facet value and see all documents. I thought facet.missing=true was
the answer.

When I set facet.missing=true in solrconfig.xml, I expected to get facet
values that are null along with their count. However, when there is no null
value, I do not want the null to be returned along with a count of zero,
which is what is happening now.

Background information: Using SolrJ with Solr 3.4 and jdk7
Sample program
SolrQuery facquery= new SolrQuery();
facquery.setQuery("*:*");
facquery.addFilterQuery("Field2:\"ISC\"");
facquery.setRows(0);
facquery.setFacet(true);
facquery.setFacetMinCount(1);
facquery.setFacetLimit(2);
String[] orderedFacetList = new String[] {"Field1", "Field2", "Field3"};
for(int i=0; i < orderedFacetList.length; i++) {
facquery.addFacetField(orderedFacetList[i]);
}
try {
facResponse = server.query(facquery);
}catch(SolrServerException ex) {
}
FacetField ff1=facResponse.getFacetField("Field2");
int count = ff1.getValueCount();//This gives count of 2
List<FacetField.Count> flist = ff1.getValues(); //The values are [ISC
(1077), null (0)]

In the above program, I am applying a filter on the field Field2 with a
value ISC. So the results will be only documents that have ISC for Field2.
My expectation is that flist in above program should only return [ISC
(1077)].

Appreciate any pointers on this. Thank you

- Rahul

Re: facet.missing=true returns null records with zero count also

[Yonik Seeley <yo...@lucidworks.com>]

On Wed, Jun 5, 2013 at 6:11 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> and think that conceptually it
> doesn't make sense for facet.missing to consider facet.mincount.

+1

"facet.missing" asks for the missing count - regardless of what it is.
Although it might make sense in some use cases to make facet.missing
pay attention to mincount, that's tougher to deal with on the client
side if it's not the behavior you want.

-Yonik
http://lucidworks.com

Re: facet.missing=true returns null records with zero count also

[Chris Hostetter <ho...@fucit.org>]

: filter again with the same facet. Also, when a facet has only one value, it
: doesn't make sense to show it to the user, since searching with that facet
: is just going to give the same result set again. So when facet.missing does
: not work with facet.mincount, it is a bit of a hassle for us.... Will work
: on handling it in our program.....Thank you for the clarification

yeah .. i totally unerstand where you are coming from, i'm just not 
certain that it's clear cut that we should change the current behavior 
since: 1) it's trivial to work arround client side; b) some other users 
might be depending on the current behavior and think that conceptually it 
doesn't make sense for facet.missing to consider facet.mincount.

i should have said before but: feel free to open an issue baout this and 
propose a patch, i'm just not sure it's a slam dunk unless we make an easy 
way to configure it to continue working the current way as well.


-Hoss

Re: facet.missing=true returns null records with zero count also

[Rahul R <ra...@gmail.com>]

Hoss,
We rely heavily on facet.mincount because once a user has selected a facet,
it doesn't make sense for us to show that facet field to him and let him
filter again with the same facet. Also, when a facet has only one value, it
doesn't make sense to show it to the user, since searching with that facet
is just going to give the same result set again. So when facet.missing does
not work with facet.mincount, it is a bit of a hassle for us.... Will work
on handling it in our program.....Thank you for the clarification

- Rahul


On Wed, Jun 5, 2013 at 12:32 AM, Chris Hostetter
<ho...@fucit.org>wrote:

>
> : that facet value and see all documents. I thought facet.missing=true was
> : the answer.
>         ...
> : facquery.setFacetMinCount(1);
>
> Hmm, yeah -- it looks like facet.missing doesn't take facet.mincount into
> consideration.
>
> I don't remember if that was intentional or not, but as a special case
> one-off count it seems like a toss up as to wether it would be more or
> less surprising to hide it if it's below the mincount. (it's very similar
> to doing one off facet.query for example, and those are always included in
> the response and don't consider the facet.mincount either)
>
> In general, this seems like a low impact thing though, correct?  i mean:
> the main advantage of facet.mincount is to reduce what could be a very
> large amount of useless data from being stream from the server->client,
> particularly in the case of using facet.sort where you really need the
> consraints eliminated server side in order to get the sort=limit applied
> correctly.
>
> but with the facet.missing value, it's just a single value per field that
> can easily be ignored by the client if it's not desired because of the
> mincount.  or to put it another way: the amount of work needed to ignor
> this on the client, is less then the amount of work to make it
> configurable to ignore it on the server.
>
>
> -Hoss
>

Re: facet.missing=true returns null records with zero count also

[Chris Hostetter <ho...@fucit.org>]

: that facet value and see all documents. I thought facet.missing=true was
: the answer.
	...
: facquery.setFacetMinCount(1);

Hmm, yeah -- it looks like facet.missing doesn't take facet.mincount into 
consideration.  

I don't remember if that was intentional or not, but as a special case 
one-off count it seems like a toss up as to wether it would be more or 
less surprising to hide it if it's below the mincount. (it's very similar 
to doing one off facet.query for example, and those are always included in 
the response and don't consider the facet.mincount either)

In general, this seems like a low impact thing though, correct?  i mean: 
the main advantage of facet.mincount is to reduce what could be a very 
large amount of useless data from being stream from the server->client, 
particularly in the case of using facet.sort where you really need the 
consraints eliminated server side in order to get the sort=limit applied 
correctly.

but with the facet.missing value, it's just a single value per field that 
can easily be ignored by the client if it's not desired because of the 
mincount.  or to put it another way: the amount of work needed to ignor 
this on the client, is less then the amount of work to make it 
configurable to ignore it on the server.


-Hoss