You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by climbingrose <cl...@gmail.com> on 2009/04/17 08:53:00 UTC
CollapseFilter with the latest Solr in trunk
Hi all,
Have any one try to use CollapseFilter with the latest version of Solr in
trunk? However, it looks like Solr 1.4 doesn't allow calling setFilterList()
and setFilter() on one instance of the QueryCommand. I modified the code in
QueryCommand to allow this:
public QueryCommand setFilterList(Query f) {
// if( filter != null ) {
// throw new IllegalArgumentException( "Either filter or filterList
may be set in the QueryCommand, but not both." );
// }
filterList = null;
if (f != null) {
filterList = new ArrayList<Query>(2);
filterList.add(f);
}
return this;
}
However, I still have a problem which prevent query filters from working
when used in conjunction with CollapseFilter. In other words, query filters
doesn't seem to have any effects on the result set when CollapseFilter is
used.
The other problem is related to OpenBitSet:
java.lang.ArrayIndexOutOfBoundsException: 2183
at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
at org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161)
at org.apache.solr.search.CollapseFilter.<init>(CollapseFilter.java:141)
at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:217)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:619)
I think CollapseFilter is rather an important function in Solr that gets
used quite frequently. Does anyone have a solution for this?
--
Regards,
Cuong Hoang
Re: CollapseFilter with the latest Solr in trunk
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Thanks for sharing!
It would be good if you (of Jeff from Zappos or anyone making changes to this) could put up a new patch for this most-voted-JIRA-issue.
Thanks,
Otis --
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
> From: climbingrose <cl...@gmail.com>
> To: solr-user@lucene.apache.org
> Sent: Sunday, April 19, 2009 8:12:11 AM
> Subject: Re: CollapseFilter with the latest Solr in trunk
>
> Ok, here is how I fixed this problem:
>
> public DocListAndSet getDocListAndSet(Query query, ListfilterList,
> DocSet docSet, Sort lsort, int offset, int len, int flags) throwsIOException {
>
> //DocListAndSet ret = new DocListAndSet();
>
> //getDocListC(ret,query,filterList,docSet,lsort,offset,len, flags |=
> GET_DOCSET);
>
> DocSet theFilt = getDocSet(filterList);
>
> if (docSet != null) theFilt = (theFilt != null) ?
> theFilt.intersection(docSet) : docSet;
>
> QueryCommand qc = new QueryCommand();
>
> qc.setQuery(query).setFilter(theFilt);
>
> qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags |=
> GET_DOCSET);
>
> QueryResult result = new QueryResult();
>
> getDocListC(result,qc);
>
>
>
> return result.getDocListAndSet();
>
> }
>
>
> There is also one-off error in CollapseFilter which you can find solution on
> Jira.
>
> Cheers,
> Cuong
>
> On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn wrote:
>
> > We are currently trying to do the same thing. With the patch unaltered we
> > can use fq as long as collapsing is turned on. If we just send a normal
> > document level query with an fq parameter it blows up.
> >
> > Additionally, it does not appear that the collapse.facet option works at
> > all.
> >
> > --
> > Jeff Newburn
> > Software Engineer, Zappos.com
> > jnewburn@zappos.com - 702-943-7562
> >
> >
> > > From: climbingrose
> > > Reply-To:
> > > Date: Fri, 17 Apr 2009 16:53:00 +1000
> > > To: solr-user
> > > Subject: CollapseFilter with the latest Solr in trunk
> > >
> > > Hi all,
> > >
> > > Have any one try to use CollapseFilter with the latest version of Solr in
> > > trunk? However, it looks like Solr 1.4 doesn't allow calling
> > setFilterList()
> > > and setFilter() on one instance of the QueryCommand. I modified the code
> > in
> > > QueryCommand to allow this:
> > >
> > > public QueryCommand setFilterList(Query f) {
> > > // if( filter != null ) {
> > > // throw new IllegalArgumentException( "Either filter or
> > filterList
> > > may be set in the QueryCommand, but not both." );
> > > // }
> > > filterList = null;
> > > if (f != null) {
> > > filterList = new ArrayList(2);
> > > filterList.add(f);
> > > }
> > > return this;
> > > }
> > >
> > > However, I still have a problem which prevent query filters from working
> > > when used in conjunction with CollapseFilter. In other words, query
> > filters
> > > doesn't seem to have any effects on the result set when CollapseFilter is
> > > used.
> > >
> > > The other problem is related to OpenBitSet:
> > >
> > > java.lang.ArrayIndexOutOfBoundsException: 2183
> > > at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
> > > at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
> > >
> > > at
> > >
> >
> >
> org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161>
> > )
> > > at
> > org.apache.solr.search.CollapseFilter.<init>(CollapseFilter.java:141)
> > >
> > > at
> > >
> > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
> > > 17)
> > > at
> > >
> > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
> > > r.java:195)
> > > at
> > >
> > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
> > > va:131)
> > >
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
> > > at
> > >
> >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303>
> > )
> > > at
> > >
> > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
> > > 2)
> > >
> > > at
> > >
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
> > > lterChain.java:202)
> > > at
> > >
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
> > > n.java:173)
> > > at
> > >
> > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
> > > :213)
> > >
> > > at
> > >
> > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
> > > :178)
> > > at
> > >
> > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> > > at
> > >
> > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> > >
> > > at
> > >
> > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
> > > 07)
> > > at
> > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> > > at
> > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> > >
> > > at
> > >
> > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processCon
> > > nection(Http11BaseProtocol.java:664)
> > > at
> > >
> > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> > > 527)
> > > at
> > >
> > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWork
> > > erThread.java:80)
> > >
> > > at
> > >
> > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
> > > 684)
> > >
> > >
> > > at java.lang.Thread.run(Thread.java:619)
> > >
> > > I think CollapseFilter is rather an important function in Solr that gets
> > > used quite frequently. Does anyone have a solution for this?
> > >
> > > --
> > > Regards,
> > >
> > > Cuong Hoang
> >
> >
>
>
> --
> Regards,
>
> Cuong Hoang
Re: CollapseFilter with the latest Solr in trunk
Posted by climbingrose <cl...@gmail.com>.
Ok, here is how I fixed this problem:
public DocListAndSet getDocListAndSet(Query query, List<Query> filterList,
DocSet docSet, Sort lsort, int offset, int len, int flags) throwsIOException {
//DocListAndSet ret = new DocListAndSet();
//getDocListC(ret,query,filterList,docSet,lsort,offset,len, flags |=
GET_DOCSET);
DocSet theFilt = getDocSet(filterList);
if (docSet != null) theFilt = (theFilt != null) ?
theFilt.intersection(docSet) : docSet;
QueryCommand qc = new QueryCommand();
qc.setQuery(query).setFilter(theFilt);
qc.setSort(lsort).setOffset(offset).setLen(len).setFlags(flags |=
GET_DOCSET);
QueryResult result = new QueryResult();
getDocListC(result,qc);
return result.getDocListAndSet();
}
There is also one-off error in CollapseFilter which you can find solution on
Jira.
Cheers,
Cuong
On Sat, Apr 18, 2009 at 4:41 AM, Jeff Newburn <jn...@zappos.com> wrote:
> We are currently trying to do the same thing. With the patch unaltered we
> can use fq as long as collapsing is turned on. If we just send a normal
> document level query with an fq parameter it blows up.
>
> Additionally, it does not appear that the collapse.facet option works at
> all.
>
> --
> Jeff Newburn
> Software Engineer, Zappos.com
> jnewburn@zappos.com - 702-943-7562
>
>
> > From: climbingrose <cl...@gmail.com>
> > Reply-To: <so...@lucene.apache.org>
> > Date: Fri, 17 Apr 2009 16:53:00 +1000
> > To: solr-user <so...@lucene.apache.org>
> > Subject: CollapseFilter with the latest Solr in trunk
> >
> > Hi all,
> >
> > Have any one try to use CollapseFilter with the latest version of Solr in
> > trunk? However, it looks like Solr 1.4 doesn't allow calling
> setFilterList()
> > and setFilter() on one instance of the QueryCommand. I modified the code
> in
> > QueryCommand to allow this:
> >
> > public QueryCommand setFilterList(Query f) {
> > // if( filter != null ) {
> > // throw new IllegalArgumentException( "Either filter or
> filterList
> > may be set in the QueryCommand, but not both." );
> > // }
> > filterList = null;
> > if (f != null) {
> > filterList = new ArrayList<Query>(2);
> > filterList.add(f);
> > }
> > return this;
> > }
> >
> > However, I still have a problem which prevent query filters from working
> > when used in conjunction with CollapseFilter. In other words, query
> filters
> > doesn't seem to have any effects on the result set when CollapseFilter is
> > used.
> >
> > The other problem is related to OpenBitSet:
> >
> > java.lang.ArrayIndexOutOfBoundsException: 2183
> > at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
> > at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
> >
> > at
> >
>
> org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161>
> )
> > at
> org.apache.solr.search.CollapseFilter.<init>(CollapseFilter.java:141)
> >
> > at
> >
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
> > 17)
> > at
> >
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
> > r.java:195)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
> > va:131)
> >
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
> > at
> >
>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303>
> )
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
> > 2)
> >
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
> > lterChain.java:202)
> > at
> >
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
> > n.java:173)
> > at
> >
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
> > :213)
> >
> > at
> >
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
> > :178)
> > at
> >
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> > at
> >
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
> >
> > at
> >
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
> > 07)
> > at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> > at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
> >
> > at
> >
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processCon
> > nection(Http11BaseProtocol.java:664)
> > at
> >
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> > 527)
> > at
> >
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWork
> > erThread.java:80)
> >
> > at
> >
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
> > 684)
> >
> >
> > at java.lang.Thread.run(Thread.java:619)
> >
> > I think CollapseFilter is rather an important function in Solr that gets
> > used quite frequently. Does anyone have a solution for this?
> >
> > --
> > Regards,
> >
> > Cuong Hoang
>
>
--
Regards,
Cuong Hoang
Re: CollapseFilter with the latest Solr in trunk
Posted by Jeff Newburn <jn...@zappos.com>.
We are currently trying to do the same thing. With the patch unaltered we
can use fq as long as collapsing is turned on. If we just send a normal
document level query with an fq parameter it blows up.
Additionally, it does not appear that the collapse.facet option works at
all.
--
Jeff Newburn
Software Engineer, Zappos.com
jnewburn@zappos.com - 702-943-7562
> From: climbingrose <cl...@gmail.com>
> Reply-To: <so...@lucene.apache.org>
> Date: Fri, 17 Apr 2009 16:53:00 +1000
> To: solr-user <so...@lucene.apache.org>
> Subject: CollapseFilter with the latest Solr in trunk
>
> Hi all,
>
> Have any one try to use CollapseFilter with the latest version of Solr in
> trunk? However, it looks like Solr 1.4 doesn't allow calling setFilterList()
> and setFilter() on one instance of the QueryCommand. I modified the code in
> QueryCommand to allow this:
>
> public QueryCommand setFilterList(Query f) {
> // if( filter != null ) {
> // throw new IllegalArgumentException( "Either filter or filterList
> may be set in the QueryCommand, but not both." );
> // }
> filterList = null;
> if (f != null) {
> filterList = new ArrayList<Query>(2);
> filterList.add(f);
> }
> return this;
> }
>
> However, I still have a problem which prevent query filters from working
> when used in conjunction with CollapseFilter. In other words, query filters
> doesn't seem to have any effects on the result set when CollapseFilter is
> used.
>
> The other problem is related to OpenBitSet:
>
> java.lang.ArrayIndexOutOfBoundsException: 2183
> at org.apache.lucene.util.OpenBitSet.fastSet(OpenBitSet.java:242)
> at org.apache.solr.search.CollapseFilter.addDoc(CollapseFilter.java:202)
>
> at
>
org.apache.solr.search.CollapseFilter.adjacentCollapse(CollapseFilter.java:161>
)
> at org.apache.solr.search.CollapseFilter.<init>(CollapseFilter.java:141)
>
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:2
> 17)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandle
> r.java:195)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.ja
> va:131)
>
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
> at
>
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303>
)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:23
> 2)
>
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFi
> lterChain.java:202)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChai
> n.java:173)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java
> :213)
>
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java
> :178)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:1
> 07)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
> at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>
> at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processCon
> nection(Http11BaseProtocol.java:664)
> at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:
> 527)
> at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWork
> erThread.java:80)
>
> at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:
> 684)
>
>
> at java.lang.Thread.run(Thread.java:619)
>
> I think CollapseFilter is rather an important function in Solr that gets
> used quite frequently. Does anyone have a solution for this?
>
> --
> Regards,
>
> Cuong Hoang