You are viewing a plain text version of this content. The canonical link for it is here.

Posted to solr-user@lucene.apache.org by Kaushik <ka...@gmail.com> on 2017/06/29 14:40:08 UTC

Number of occurrences in Solr Documents

Hello,

We are trying to get the most frequently used words in a collection. My
understanding is that using facet.field=content_txt. An e.g. of content_txt
value is "The fox jumped over another fox". In such a scenario, I am
expecting the facet to return with "fox" and with a count value of 2.
However, we end up getting "fox" with a value of 1. It appears we are
getting total number of documents that match the query as opposed to the
total number of times the word ocurred. How can the latter be achieved?

Thanks,
AK

Re: Number of occurrences in Solr Documents

Posted by Susheel Kumar <su...@gmail.com>.

Checkout Term Vector component

https://wiki.apache.org/solr/TermVectorComponent

On Thu, Jun 29, 2017 at 10:40 AM, Kaushik <ka...@gmail.com> wrote:

> Hello,
>
> We are trying to get the most frequently used words in a collection. My
> understanding is that using facet.field=content_txt. An e.g. of content_txt
> value is "The fox jumped over another fox". In such a scenario, I am
> expecting the facet to return with "fox" and with a count value of 2.
> However, we end up getting "fox" with a value of 1. It appears we are
> getting total number of documents that match the query as opposed to the
> total number of times the word ocurred. How can the latter be achieved?
>
> Thanks,
> AK
>

Re: Number of occurrences in Solr Documents

Posted by David Hastings <ha...@gmail.com>.

I am using 5.2 and this works:

select?q=*%3A*&wt=csv&indent=true&fl=totaltermfreq(text%2WORDIWANTTOFIND)&rows=1

On Thu, Jun 29, 2017 at 11:52 AM, Kaushik <ka...@gmail.com> wrote:

> Thanks to Susheel and Shawn. Unfortunately the Solr version we have is Solr
> 5.3 and it does not include the totaltermfrequency feature. Is there any
> downside of using TermVectorFrequency ; like peformance issues?
>
> On Thu, Jun 29, 2017 at 11:49 AM, Susheel Kumar <su...@gmail.com>
> wrote:
>
> > That's even better. Thanks, Shawn.
> >
> > On Thu, Jun 29, 2017 at 11:45 AM, Shawn Heisey <ap...@elyograg.org>
> > wrote:
> >
> > > On 6/29/2017 8:40 AM, Kaushik wrote:
> > > > We are trying to get the most frequently used words in a collection.
> > > > My understanding is that using facet.field=content_txt. An e.g. of
> > > > content_txt value is "The fox jumped over another fox". In such a
> > > > scenario, I am expecting the facet to return with "fox" and with a
> > > > count value of 2. However, we end up getting "fox" with a value of 1.
> > > > It appears we are getting total number of documents that match the
> > > > query as opposed to the total number of times the word ocurred. How
> > > > can the latter be achieved?
> > >
> > > Facets count the number of documents, not the number of terms.
> > >
> > > You might be after the terms component.
> > >
> > > https://lucene.apache.org/solr/guide/6_6/the-terms-component.html
> > >
> > > This generally works across the entire index, while facets can operate
> > > on documents that match a query.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>

Re: Number of occurrences in Solr Documents

Posted by Kaushik <ka...@gmail.com>.

Thanks to Susheel and Shawn. Unfortunately the Solr version we have is Solr
5.3 and it does not include the totaltermfrequency feature. Is there any
downside of using TermVectorFrequency ; like peformance issues?

On Thu, Jun 29, 2017 at 11:49 AM, Susheel Kumar <su...@gmail.com>
wrote:

> That's even better. Thanks, Shawn.
>
> On Thu, Jun 29, 2017 at 11:45 AM, Shawn Heisey <ap...@elyograg.org>
> wrote:
>
> > On 6/29/2017 8:40 AM, Kaushik wrote:
> > > We are trying to get the most frequently used words in a collection.
> > > My understanding is that using facet.field=content_txt. An e.g. of
> > > content_txt value is "The fox jumped over another fox". In such a
> > > scenario, I am expecting the facet to return with "fox" and with a
> > > count value of 2. However, we end up getting "fox" with a value of 1.
> > > It appears we are getting total number of documents that match the
> > > query as opposed to the total number of times the word ocurred. How
> > > can the latter be achieved?
> >
> > Facets count the number of documents, not the number of terms.
> >
> > You might be after the terms component.
> >
> > https://lucene.apache.org/solr/guide/6_6/the-terms-component.html
> >
> > This generally works across the entire index, while facets can operate
> > on documents that match a query.
> >
> > Thanks,
> > Shawn
> >
> >
>

Re: Number of occurrences in Solr Documents

Posted by Susheel Kumar <su...@gmail.com>.

That's even better. Thanks, Shawn.

On Thu, Jun 29, 2017 at 11:45 AM, Shawn Heisey <ap...@elyograg.org> wrote:

> On 6/29/2017 8:40 AM, Kaushik wrote:
> > We are trying to get the most frequently used words in a collection.
> > My understanding is that using facet.field=content_txt. An e.g. of
> > content_txt value is "The fox jumped over another fox". In such a
> > scenario, I am expecting the facet to return with "fox" and with a
> > count value of 2. However, we end up getting "fox" with a value of 1.
> > It appears we are getting total number of documents that match the
> > query as opposed to the total number of times the word ocurred. How
> > can the latter be achieved?
>
> Facets count the number of documents, not the number of terms.
>
> You might be after the terms component.
>
> https://lucene.apache.org/solr/guide/6_6/the-terms-component.html
>
> This generally works across the entire index, while facets can operate
> on documents that match a query.
>
> Thanks,
> Shawn
>
>

Re: Number of occurrences in Solr Documents

Posted by Shawn Heisey <ap...@elyograg.org>.

On 6/29/2017 8:40 AM, Kaushik wrote:
> We are trying to get the most frequently used words in a collection.
> My understanding is that using facet.field=content_txt. An e.g. of
> content_txt value is "The fox jumped over another fox". In such a
> scenario, I am expecting the facet to return with "fox" and with a
> count value of 2. However, we end up getting "fox" with a value of 1.
> It appears we are getting total number of documents that match the
> query as opposed to the total number of times the word ocurred. How
> can the latter be achieved?

Facets count the number of documents, not the number of terms.

You might be after the terms component.

https://lucene.apache.org/solr/guide/6_6/the-terms-component.html

This generally works across the entire index, while facets can operate
on documents that match a query.

Thanks,
Shawn