You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Steve Molloy (JIRA)" <ji...@apache.org> on 2014/07/06 21:44:36 UTC
[jira] [Commented] (SOLR-3583) Percentiles for facets, pivot
facets, and distributed pivot facets
[ https://issues.apache.org/jira/browse/SOLR-3583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053208#comment-14053208 ]
Steve Molloy commented on SOLR-3583:
------------------------------------
Tried applying patch with latest SOLR-2894 patch (2014-07-02) on top of 4.9 and get a failure in the unit test. Already switched the commented lines around dates (using parsed dates instead of strings) and that helps. Bute no it seems to not be getting same results on single shard and distributed. I get the following output when running ant test for it:
[junit4] 2> 7009 T57 C8 oasc.SolrCore.execute [collection1] webapp= path=/select params={f.hourlypay_d.percentiles.gap=1.0&facet.field=hiredate_dt&percentiles=true&f.hourlypay_d.percentiles_buckets=true&f.hourlypay_d.percentiles.upper.fence=150.0&f.yearlypay_ti.percentiles_buckets=true&facet.pivot=company_s,hiredate_dt&percentiles.requested.percentiles=25,50,75,80,91,96,97,99&f.yearlypay_ti.percentiles.lower.fence=5000&f.yearlypay_ti.percentiles.gap=1000&f.hiredate_dt.percentiles.upper.fence=2012-07-30T00:00:00Z&f.hourlypay_d.percentiles.lower.fence=5.50&rows=0&percentiles.field=yearlypay_ti&percentiles.field=hourlypay_d&percentiles.field=hiredate_dt&version=2&q=*:*&shards=[ff01::114]:33332/|[ff01::213]:33332/|https://127.0.0.1:57329/|[ff01::083]:33332/,https://127.0.0.1:36083/&f.yearlypay_ti.percentiles.upper.fence=500000&f.hiredate_dt.percentiles.gap=%2B1DAYS&f.hiredate_dt.percentiles.lower.fence=2012-05-01T00:00:00Z&f.hiredate_dt.percentiles_buckets=true&facet=true&wt=javabin} hits=10 status=0 QTime=44
[junit4] 2> 7019 T12 oas.BaseDistributedSearchTestCase.compareSolrResponses ERROR Mismatched responses:
[junit4] 2> {responseHeader={status=0,QTime=44},response={numFound=10,start=0,maxScore=1.0,docs=[]},facet_counts={facet_queries={},facet_fields={hiredate_dt={2012-07-01T12:30:00Z=10}},facet_dates={},facet_ranges={},facet_pivot={company_s,hiredate_dt=[{field=company_s,value=Stark Industries,count=10,statistics={yearlypay_ti={percentiles={25.0=31500,50.0=35500,75.0=37500,80.0=37500,91.0=37500,96.0=37500,97.0=37500,99.0=37500}},hourlypay_d={percentiles={25.0=91.0,50.0=95.0,75.0=95.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}},pivot=[{field=hiredate_dt,value=Sun Jul 01 08:30:00 EDT 2012,count=10,statistics={yearlypay_ti={percentiles={25.0=31500,50.0=35500,75.0=37500,80.0=37500,91.0=37500,96.0=37500,97.0=37500,99.0=37500}},hourlypay_d={percentiles={25.0=91.0,50.0=95.0,75.0=95.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}}}]}]}}}
[junit4] 2> {responseHeader={status=0,QTime=6},response={numFound=10,start=0,docs=[]},facet_counts={facet_queries={},facet_fields={hiredate_dt={2012-07-01T12:30:00Z=10}},facet_dates={},facet_ranges={},facet_pivot={company_s,hiredate_dt=[{field=company_s,value=Stark Industries,count=10,statistics={yearlypay_ti={percentiles={25.0=32500,50.0=35500,75.0=37500,80.0=37500,91.0=38500,96.0=38500,97.0=38500,99.0=38500}},hourlypay_d={percentiles={25.0=95.0,50.0=95.0,75.0=100.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}},pivot=[{field=hiredate_dt,value=Sun Jul 01 08:30:00 EDT 2012,count=10,statistics={yearlypay_ti={percentiles={25.0=32500,50.0=35500,75.0=37500,80.0=37500,91.0=38500,96.0=38500,97.0=38500,99.0=38500}},hourlypay_d={percentiles={25.0=95.0,50.0=95.0,75.0=100.0,80.0=100.0,91.0=100.0,96.0=100.0,97.0=100.0,99.0=100.0}},hiredate_dt={percentiles={25.0=2012-07-01T12:00:00Z,50.0=2012-07-01T12:00:00Z,75.0=2012-07-01T12:00:00Z,80.0=2012-07-01T12:00:00Z,91.0=2012-07-01T12:00:00Z,96.0=2012-07-01T12:00:00Z,97.0=2012-07-01T12:00:00Z,99.0=2012-07-01T12:00:00Z}}}}]}]}}}
[junit4] 2> 7044 T12 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1953253883
[junit4] 2> 7045 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@25dc8c85
[junit4] 2> 7045 T12 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=40,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=4755,transaction_logs_total_number=4}
[junit4] 2> 7046 T12 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 7046 T12 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7046 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 7047 T12 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 7047 T12 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index;done=false>>]
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data/index
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data;done=false>>]
[junit4] 2> 7048 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/control/data
[junit4] 2> 7049 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7123 T12 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1841909798
[junit4] 2> 7123 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@35e21988
[junit4] 2> 7124 T12 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=22,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=2719,transaction_logs_total_number=4}
[junit4] 2> 7124 T12 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 7124 T12 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7124 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 7125 T12 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data;done=false>>]
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index;done=false>>]
[junit4] 2> 7126 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard0/data/index
[junit4] 2> 7127 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7199 T12 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1421481657
[junit4] 2> 7200 T12 oasc.SolrCore.close [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2e4c44b2
[junit4] 2> 7200 T12 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=18,cumulative_deletesById=0,cumulative_deletesByQuery=2,cumulative_errors=0,transaction_logs_total_size=2376,transaction_logs_total_number=4}
[junit4] 2> 7205 T12 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4] 2> 7205 T12 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4] 2> 7205 T12 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4] 2> 7206 T12 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index;done=false>>]
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data/index
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data [CachedDir<<refCount=0;path=./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data;done=false>>]
[junit4] 2> 7207 T12 oasc.CachingDirectoryFactory.close Closing directory: ./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001/tempDir-001/shard1/data
[junit4] 2> 7208 T12 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4] 2> 7259 T12 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=DistributedFacetPivotStatisticsTest -Dtests.method=testDistribSearch -Dtests.seed=CD32BB6D12588826 -Dtests.slow=true -Dtests.locale=ms -Dtests.timezone=America/Indiana/Indianapolis -Dtests.file.encoding=ISO-8859-1
[junit4] FAILURE 6.82s | DistributedFacetPivotStatisticsTest.testDistribSearch <<<
[junit4] > Throwable #1: junit.framework.AssertionFailedError: .facet_counts.facet_pivot.company_s,hiredate_dt[0].statistics.yearlypay_ti.percentiles.25.0:31500!=32500
[junit4] > at __randomizedtesting.SeedInfo.seed([CD32BB6D12588826:4CD435756507E81A]:0)
[junit4] > at junit.framework.Assert.fail(Assert.java:50)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.compareSolrResponses(BaseDistributedSearchTestCase.java:838)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.compareResponses(BaseDistributedSearchTestCase.java:857)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:560)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:540)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:519)
[junit4] > at org.apache.solr.handler.component.DistributedFacetPivotStatisticsTest.doTest(DistributedFacetPivotStatisticsTest.java:430)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:871)
[junit4] > at java.lang.Thread.run(Thread.java:745)
[junit4] 2> 7282 T12 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4] 2> NOTE: leaving temporary files on disk at: /home/smolloy/workspace-solr/otif-trunk/solr/build/solr-core/test/J0/./solr.handler.component.DistributedFacetPivotStatisticsTest-CD32BB6D12588826-001
[junit4] 2> NOTE: test params are: codec=Lucene49: {hiredate_dt=Pulsing41(freqCutoff=19 minBlockSize=100 maxBlockSize=291), _version_=FSTOrd41, multiDefault=FST41, yearlypay_ti=FSTOrd41, intDefault=FSTOrd41, range_facet_sl=Pulsing41(freqCutoff=19 minBlockSize=100 maxBlockSize=291), range_facet_si=FST41, company_s=PostingsFormat(name=Lucene41WithOrds), id=PostingsFormat(name=Lucene41WithOrds), hourlypay_d=FST41, range_facet_l=FSTOrd41, timestamp=FSTOrd41}, docValues:{}, sim=DefaultSimilarity, locale=ms, timezone=America/Indiana/Indianapolis
[junit4] 2> NOTE: Linux 3.13.0-30-generic amd64/Oracle Corporation 1.8.0_05 (64-bit)/cpus=8,threads=1,free=170392096,total=206045184
[junit4] 2> NOTE: All tests run in this JVM: [DistributedFacetPivotStatisticsTest]
[junit4] Completed in 7.38s, 1 test, 1 failure <<< FAILURES!
[junit4]
[junit4] JVM J0: stderr was not empty, see: /home/smolloy/workspace-solr/otif-trunk/solr/build/solr-core/test/temp/junit4-J0-20140706_153759_130.syserr
[junit4] >>> JVM J0: stderr (verbatim) ----
[junit4] Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128m; support was removed in 8.0
[junit4] <<< JVM J0: EOF ----
[junit4]
[junit4] Tests with failures:
[junit4] - org.apache.solr.handler.component.DistributedFacetPivotStatisticsTest.testDistribSearch
[junit4]
[junit4]
[junit4] JVM J0: 0.62 .. 8.67 = 8.06s
[junit4] Execution time total: 8.68 sec.
[junit4] Tests summary: 1 suite, 1 test, 1 failure
Seems some values are not considered in distributed response. Do you have pointers where to look for cause of differences?
> Percentiles for facets, pivot facets, and distributed pivot facets
> ------------------------------------------------------------------
>
> Key: SOLR-3583
> URL: https://issues.apache.org/jira/browse/SOLR-3583
> Project: Solr
> Issue Type: Improvement
> Reporter: Chris Russell
> Priority: Minor
> Labels: newbie, patch
> Fix For: 4.9, 5.0
>
> Attachments: SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch, SOLR-3583.patch
>
>
> Built on top of SOLR-2894, this patch adds percentiles and averages to facets, pivot facets, and distributed pivot facets by making use of range facet internals.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org