docFreq coming to be more than 1 for unique id field

[Apoorva Gaurav <ap...@myntra.com>]

Hello All,

We are using solr 4.4.0. We have a uniqueKey of type solr.StrField. We need
to extract docs in a pre-defined order if they match a certain condition.
Our query is of the format

uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
where weight1 > weight2 > ........ > weightN

But the result is not in the desired order. On debugging the query we've
found out that for some of the documents docFreq is higher than 1 and hence
their tf-idf based score is less than others. What can be the reason behind
a unique id field having docFreq greater than 1?  How can we prevent it?

-- 
Thanks & Regards,
Apoorva

Re: docFreq coming to be more than 1 for unique id field

[Apoorva Gaurav <ap...@myntra.com>]

Hello Markus, Ahmet,
Forgot to update the thread; optimization works i.e. after optimizing all
unique keys have docFreq as 1.


On Wed, Jun 18, 2014 at 1:58 AM, Chris Hostetter <ho...@fucit.org>
wrote:

>
> : text in it, query is of the type "keywords:(word1 OR word2 ... OR
> wordN)".
> : The client is relying on default relevancy based sort returned by solr.
> : Some documents can get penalised because of some other documents which
> were
> : deleted. Is this functionality correct?
>
> yes, because term stats are over the entire index including deleted
> documents still in segments -- information about deletions isn't "purged"
> from the index until a segment is merged and the stats are recomputed over
> the docs/terms in the new segment.
>
> the only way to get those types of statistics at request time such that
> they were *not* afected by deleted documents would involve scanning every
> doc to compute them -- which would defeat the point of having the inverted
> index.
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
Thanks & Regards,
Apoorva

Re: docFreq coming to be more than 1 for unique id field

[Chris Hostetter <ho...@fucit.org>]

: text in it, query is of the type "keywords:(word1 OR word2 ... OR wordN)".
: The client is relying on default relevancy based sort returned by solr.
: Some documents can get penalised because of some other documents which were
: deleted. Is this functionality correct?

yes, because term stats are over the entire index including deleted 
documents still in segments -- information about deletions isn't "purged" 
from the index until a segment is merged and the stats are recomputed over 
the docs/terms in the new segment.

the only way to get those types of statistics at request time such that 
they were *not* afected by deleted documents would involve scanning every 
doc to compute them -- which would defeat the point of having the inverted 
index.


-Hoss
http://www.lucidworks.com/

Re: docFreq coming to be more than 1 for unique id field

[Apoorva Gaurav <ap...@myntra.com>]

OK lets for a moment forget about this specific use case and consider a
more general case. Lets say the field name is "keywords" are we are storing
text in it, query is of the type "keywords:(word1 OR word2 ... OR wordN)".
The client is relying on default relevancy based sort returned by solr.
Some documents can get penalised because of some other documents which were
deleted. Is this functionality correct?


On Wed, Jun 18, 2014 at 12:52 AM, Chris Hostetter <ho...@fucit.org>
wrote:

>
> : Currently we are not using SolrJ but are simply interacting with solr
> with
> : json over http, this will change in a couple of months but currently not
> : there. As of now we are putting all the logic in query building, using it
> : to query solr and then passing on the json returned by it to front end. I
> : know this is not the ideal approach, but that's what we have at the
> moment.
> : Hence need a way of deterministically order the result set provided they
> : match other search criteria.
>
> wether you are using SOlrJ or not doesn't really change my point at all --
> you are jumping though all sorts of hoops, and asking solr to jump through
> all sorts of hoops, for a score you don't actaully care about, and isn't
> going ot work perfectly for what you want anyway because of the
> fundemental nature of the inverted index stats, leading you to look for
> even smaller, higher, hoops to try and jump through.
>
> it would be far simpler to just ask for the exact set of N documents you
> wnat from Solr in default order, re-order the resulting documents in the
> magic order you already know and care about, and then give that modified
> response to your front end.
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
Thanks & Regards,
Apoorva

Re: docFreq coming to be more than 1 for unique id field

[Chris Hostetter <ho...@fucit.org>]

: Currently we are not using SolrJ but are simply interacting with solr with
: json over http, this will change in a couple of months but currently not
: there. As of now we are putting all the logic in query building, using it
: to query solr and then passing on the json returned by it to front end. I
: know this is not the ideal approach, but that's what we have at the moment.
: Hence need a way of deterministically order the result set provided they
: match other search criteria.

wether you are using SOlrJ or not doesn't really change my point at all -- 
you are jumping though all sorts of hoops, and asking solr to jump through 
all sorts of hoops, for a score you don't actaully care about, and isn't 
going ot work perfectly for what you want anyway because of the 
fundemental nature of the inverted index stats, leading you to look for 
even smaller, higher, hoops to try and jump through.

it would be far simpler to just ask for the exact set of N documents you 
wnat from Solr in default order, re-order the resulting documents in the 
magic order you already know and care about, and then give that modified 
response to your front end.


-Hoss
http://www.lucidworks.com/

Re: docFreq coming to be more than 1 for unique id field

[Apoorva Gaurav <ap...@myntra.com>]

Currently we are not using SolrJ but are simply interacting with solr with
json over http, this will change in a couple of months but currently not
there. As of now we are putting all the logic in query building, using it
to query solr and then passing on the json returned by it to front end. I
know this is not the ideal approach, but that's what we have at the moment.
Hence need a way of deterministically order the result set provided they
match other search criteria.


On Tue, Jun 17, 2014 at 10:28 PM, Chris Hostetter <ho...@fucit.org>
wrote:

>
> All index wide statistics (like the docFreq of each term) are over the
> entire index, which includes deleted docs -- because it's an *inverted*
> index, it's not feasible to update those statistics to account for deleted
> docs (that would basically kill all the performance advantages thatcome
> from having an inverted index.
>
>
> : uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
> : where weight1 > weight2 > ........ > weightN
> :
> : But the result is not in the desired order. On debugging the query we've
>
> if you are requesting a small number of docs, and all the docs you are
> requesting are returned in a single request, why do you care what order
> they are in?  why not just put them in hte order you want on the client.
>
> That would not only make your solr request simpler, but would almost
> certainly be a bit *faster* since you could sort exactly as you wnated w/o
> needing to compute a complex score that you don't actaully care about.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>



-- 
Thanks & Regards,
Apoorva

Re: docFreq coming to be more than 1 for unique id field

[Chris Hostetter <ho...@fucit.org>]

All index wide statistics (like the docFreq of each term) are over the 
entire index, which includes deleted docs -- because it's an *inverted* 
index, it's not feasible to update those statistics to account for deleted 
docs (that would basically kill all the performance advantages thatcome 
from having an inverted index.


: uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
: where weight1 > weight2 > ........ > weightN
: 
: But the result is not in the desired order. On debugging the query we've

if you are requesting a small number of docs, and all the docs you are 
requesting are returned in a single request, why do you care what order 
they are in?  why not just put them in hte order you want on the client.

That would not only make your solr request simpler, but would almost 
certainly be a bit *faster* since you could sort exactly as you wnated w/o 
needing to compute a complex score that you don't actaully care about.



-Hoss
http://www.lucidworks.com/

RE: docFreq coming to be more than 1 for unique id field

[Markus Jelsma <ma...@openindex.io>]

Hi - did you perhaps update on of those documents?

 
 
-----Original message-----
> From:Apoorva Gaurav <ap...@myntra.com>
> Sent: Tuesday 17th June 2014 16:58
> To: solr-user@lucene.apache.org
> Subject: docFreq coming to be more than 1 for unique id field
> 
> Hello All,
> 
> We are using solr 4.4.0. We have a uniqueKey of type solr.StrField. We need
> to extract docs in a pre-defined order if they match a certain condition.
> Our query is of the format
> 
> uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
> where weight1 > weight2 > ........ > weightN
> 
> But the result is not in the desired order. On debugging the query we've
> found out that for some of the documents docFreq is higher than 1 and hence
> their tf-idf based score is less than others. What can be the reason behind
> a unique id field having docFreq greater than 1?  How can we prevent it?
> 
> -- 
> Thanks & Regards,
> Apoorva
> 

Re: docFreq coming to be more than 1 for unique id field

[Jack Krupansky <ja...@basetechnology.com>]

Personally, although I understand the rationale and performance 
ramifications of the current approach of including deleted documents, I 
would agree that DF and IDF should definitely be accurate, despite 
deletions. So, if they aren't, I'd suggest filing a bug Jira. Granted it 
might be rejected as "by design" or "won't fix" or "improvement", but it's 
worth having the discussion.

Maybe one theory from the old days is that the model of "batch update" would 
by definition include an optimize step. But now with Solr considered by some 
to be a "NoSQL database" and with (near) real-time updates, that model is 
clearly obsolete.

-- Jack Krupansky

-----Original Message----- 
From: Apoorva Gaurav
Sent: Tuesday, June 17, 2014 11:15 AM
To: solr-user ; Ahmet Arslan
Subject: Re: docFreq coming to be more than 1 for unique id field

Yes we have updates on these. Didn't try optimizing will do. But isn't the
unique field supposed to be unique?


On Tue, Jun 17, 2014 at 8:37 PM, Ahmet Arslan <io...@yahoo.com.invalid>
wrote:

> Hi,
>
> Just a guess, do you have deletions? What happens when you optimize and
> re-try?
>
>
>
> On Tuesday, June 17, 2014 5:58 PM, Apoorva Gaurav <
> apoorva.gaurav@myntra.com> wrote:
> Hello All,
>
> We are using solr 4.4.0. We have a uniqueKey of type solr.StrField. We 
> need
> to extract docs in a pre-defined order if they match a certain condition.
> Our query is of the format
>
> uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
> where weight1 > weight2 > ........ > weightN
>
> But the result is not in the desired order. On debugging the query we've
> found out that for some of the documents docFreq is higher than 1 and 
> hence
> their tf-idf based score is less than others. What can be the reason 
> behind
> a unique id field having docFreq greater than 1?  How can we prevent it?
>
> --
> Thanks & Regards,
> Apoorva
>
>


-- 
Thanks & Regards,
Apoorva 

Re: docFreq coming to be more than 1 for unique id field

[Apoorva Gaurav <ap...@myntra.com>]

Yes we have updates on these. Didn't try optimizing will do. But isn't the
unique field supposed to be unique?


On Tue, Jun 17, 2014 at 8:37 PM, Ahmet Arslan <io...@yahoo.com.invalid>
wrote:

> Hi,
>
> Just a guess, do you have deletions? What happens when you optimize and
> re-try?
>
>
>
> On Tuesday, June 17, 2014 5:58 PM, Apoorva Gaurav <
> apoorva.gaurav@myntra.com> wrote:
> Hello All,
>
> We are using solr 4.4.0. We have a uniqueKey of type solr.StrField. We need
> to extract docs in a pre-defined order if they match a certain condition.
> Our query is of the format
>
> uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
> where weight1 > weight2 > ........ > weightN
>
> But the result is not in the desired order. On debugging the query we've
> found out that for some of the documents docFreq is higher than 1 and hence
> their tf-idf based score is less than others. What can be the reason behind
> a unique id field having docFreq greater than 1?  How can we prevent it?
>
> --
> Thanks & Regards,
> Apoorva
>
>


-- 
Thanks & Regards,
Apoorva

Re: docFreq coming to be more than 1 for unique id field

[Ahmet Arslan <io...@yahoo.com.INVALID>]

Hi,

Just a guess, do you have deletions? What happens when you optimize and re-try?



On Tuesday, June 17, 2014 5:58 PM, Apoorva Gaurav <ap...@myntra.com> wrote:
Hello All,

We are using solr 4.4.0. We have a uniqueKey of type solr.StrField. We need
to extract docs in a pre-defined order if they match a certain condition.
Our query is of the format

uniqueField:(id1 ^ weight1 OR id2 ^ weight2 ..... OR idN ^ weightN)
where weight1 > weight2 > ........ > weightN

But the result is not in the desired order. On debugging the query we've
found out that for some of the documents docFreq is higher than 1 and hence
their tf-idf based score is less than others. What can be the reason behind
a unique id field having docFreq greater than 1?  How can we prevent it?

-- 
Thanks & Regards,
Apoorva