Term Frequencies for Query Result

[Kai Gülzau <kg...@novomind.com>]

Hi,

I am looking for a way to get the top terms for a query result.

Faceting does not work since counts are measured as documents containing a term and not as the overall count of a term in all found documents:

http://localhost:8983/solr/master/select?q=type%3A7&rows=1&wt=json&indent=true&facet=true&facet.query=type%3A7&facet.field=albody&facet.method=fc

  "facet_counts":{
    "facet_queries":{
      "type:7":156},
    "facet_fields":{
      "albody":[
        "der",73,
        "in",68,
        "betreff",63,
...


Using http://wiki.apache.org/solr/TermVectorComponent an counting all frequencies manually seems to be the only solution by now:

http://localhost:8983/solr/tvrh/?q=type:7&tv.fl=albody&f.albody.tv.tf=true&wt=json&indent=true


"termVectors":[

    "uniqueKeyFieldName","ukey",

    "798_7_0",[

      "uniqueKey","798_7_0",

      "albody",[

        "der",[

          "tf",5],

        "die",[

          "tf",7],

...



Does anyone know a better and more efficient solution?


Regards,

Kai Gülzau

RE: Term Frequencies for Query Result

[Kai Gülzau <kg...@novomind.com>]

> i *think* you are saying that you want the sum of term frequencies for all 
> terms in all matching documents -- but i'm not sure, because i don't see 
> how TermVectorComponent is helping you unless you are iterating over every 
> doc in the result set (ie: deep paging) to get the TermVectors for every 
> doc ... it would help if you could explain what you mean by "counting all 
> frequencies manually"

You are good in guessing :-)
Saying "counting all frequencies manually" I think of collecting term
frequencies for each term while iterating over all documents.


>> I am looking for a way to get the top terms for a query result.
> you have to elaborate on exactly what you mean ... how are you defining 
> "top terms for a query result" ?  Are you talking about the most common 
> terms in the entire result set of documents that match your query?

My goal is to show the most relevant keywords for some documents of the index.
So "top terms for a query result" should be "top nouns for a filtered query".

While using faceting "top" means "sorted by count of docs containing the term".

When I could get the sum of the term frequencies, my hope is to be able
to distinguish between too common terms and more relevant terms.
Something like a score for a term based on a filtered query.


regards,

Kai Gülzau

Re: Term Frequencies for Query Result

[Chris Hostetter <ho...@fucit.org>]

: I am looking for a way to get the top terms for a query result.

you have to elaborate on exactly what you mean ... how are you defining 
"top terms for a query result" ?  Are you talking about hte most common 
terms in the entire result set of documents that match your query?  or the 
terms from the query that most contributed to the query? or something 
else?

: Faceting does not work since counts are measured as documents containing 
: a term and not as the overall count of a term in all found documents:
	...
: Using http://wiki.apache.org/solr/TermVectorComponent an counting all 
: frequencies manually seems to be the only solution by now:

i *think* you are saying that you want the sum of term frequencies for all 
terms in all matching documents -- but i'm not sure, because i don't see 
how TermVectorComponent is helping you unless you are iterating over every 
doc in the result set (ie: deep paging) to get the TermVectors for every 
doc ... it would help if you could explain what you mean by "counting all 
frequencies manually"



-Hoss