You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Karl Wettin (JIRA)" <ji...@apache.org> on 2006/07/22 21:50:14 UTC
[jira] Updated: (LUCENE-550) InstanciatedIndex - faster but memory
consuming index
[ http://issues.apache.org/jira/browse/LUCENE-550?page=all ]
Karl Wettin updated LUCENE-550:
-------------------------------
Attachment: lucene2-karl_20060722.tar.gz
New code. More backwards compatible. Just a very few changes required to the Lucene core.
Now with test cases from distribution, but only search/* has been ported. Fails some (11 of 172) score and RMI related tests that I can not explain. Could really need some help with that
Except for that this seems to work really great now. I've been running this in a live environment for a few hours (some hundred thousand user queries) and it is *really* fast.
Output from failing tests:
junit.framework.AssertionFailedError: expected:<3> but was:<0>
at org.apache.lucene.search.TestPhraseQuery.testSlopScoring(TestPhraseQuery.java:298)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
junit.framework.AssertionFailedError: Using 10 documents per index:
at org.apache.lucene.search.TestMultiSearcher.testNormalization(TestMultiSearcher.java:247)
at org.apache.lucene.search.TestMultiSearcher.testNormalization10(TestMultiSearcher.java:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
------- testSimpleEqualScores1 -------
#0: 1.000000000 - d3
#1: 1.000000000 - d4
#2: 0.500000000 - d1
#3: 0.500000000 - d2
junit.framework.AssertionFailedError: score #2 is not the same expected:<1.0> but was:<0.5>
at org.apache.lucene.search.TestDisjunctionMaxQuery.testSimpleEqualScores1(TestDisjunctionMaxQuery.java:142)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
------- testSimpleEqualScores2 -------
#0: 1.000000000 - d2
#1: 0.500000000 - d1
#2: 0.500000000 - d4
junit.framework.AssertionFailedError: score #1 is not the same expected:<1.0> but was:<0.5>
at org.apache.lucene.search.TestDisjunctionMaxQuery.testSimpleEqualScores2(TestDisjunctionMaxQuery.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
------- testSimpleEqualScores3 -------
#0: 1.000000000 - d2
#1: 1.000000000 - d3
#2: 1.000000000 - d4
#3: 0.500000000 - d1
junit.framework.AssertionFailedError: score #3 is not the same expected:<1.0> but was:<0.5>
at org.apache.lucene.search.TestDisjunctionMaxQuery.testSimpleEqualScores3(TestDisjunctionMaxQuery.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
junit.framework.AssertionFailedError: A,B,D, only B in range expected:<1> but was:<2>
at org.apache.lucene.search.TestRangeQuery.testExclusive(TestRangeQuery.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
junit.framework.AssertionFailedError: A,B,D - A and B in range expected:<2> but was:<5>
at org.apache.lucene.search.TestRangeQuery.testInclusive(TestRangeQuery.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
junit.framework.AssertionFailedError: Using 10 documents per index:
at org.apache.lucene.search.TestMultiSearcher.testNormalization(TestMultiSearcher.java:247)
at org.apache.lucene.search.TestMultiSearcher.testNormalization10(TestMultiSearcher.java:220)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
java.rmi.server.ExportException: internal error: ObjID already in use
at sun.rmi.transport.ObjectTable.putTarget(ObjectTable.java:197)
at sun.rmi.transport.Transport.exportObject(Transport.java:90)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:231)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:398)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:131)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:195)
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:107)
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:93)
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:198)
at org.apache.lucene.search.TestSort.startServer(TestSort.java:704)
at org.apache.lucene.search.TestSort.getRemote(TestSort.java:689)
at org.apache.lucene.search.TestSort.testRemoteSort(TestSort.java:410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
java.rmi.server.ExportException: internal error: ObjID already in use
at sun.rmi.transport.ObjectTable.putTarget(ObjectTable.java:197)
at sun.rmi.transport.Transport.exportObject(Transport.java:90)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:231)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:398)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:131)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:195)
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:107)
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:93)
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:198)
at org.apache.lucene.search.TestSort.startServer(TestSort.java:704)
at org.apache.lucene.search.TestSort.getRemote(TestSort.java:689)
at org.apache.lucene.search.TestSort.testRemoteCustomSort(TestSort.java:417)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
java.rmi.server.ExportException: internal error: ObjID already in use
at sun.rmi.transport.ObjectTable.putTarget(ObjectTable.java:197)
at sun.rmi.transport.Transport.exportObject(Transport.java:90)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:231)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:398)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:131)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:195)
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:107)
at sun.rmi.registry.RegistryImpl.<init>(RegistryImpl.java:93)
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:198)
at org.apache.lucene.search.TestSort.startServer(TestSort.java:704)
at org.apache.lucene.search.TestSort.getRemote(TestSort.java:689)
at org.apache.lucene.search.TestSort.testNormalizedScores(TestSort.java:440)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.junit2.JUnitStarter.main(JUnitStarter.java:32)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:90)
> InstanciatedIndex - faster but memory consuming index
> -----------------------------------------------------
>
> Key: LUCENE-550
> URL: http://issues.apache.org/jira/browse/LUCENE-550
> Project: Lucene - Java
> Issue Type: New Feature
> Components: Store
> Affects Versions: 1.9
> Reporter: Karl Wettin
> Attachments: class_diagram.png, class_diagram.png, instanciated_20060527.tar, InstanciatedIndexTermEnum.java, lucene.1.9-karl1.jpg, lucene2-karl_20060722.tar.gz
>
>
> After fixing the bugs, it's now 4.5 -> 5 times the speed. This is true for both at index and query time. Sorry if I got your hopes up too much. There are still things to be done though. Might not have time to do anything with this until next month, so here is the code if anyone wants a peek.
> Not good enough for Jira yet, but if someone wants to fool around with it, here it is. The implementation passes a TermEnum -> TermDocs -> Fields -> TermVector comparation against the same data in a Directory.
> When it comes to features, offsets don't exists and positions are stored ugly and has bugs.
> You might notice that norms are float[] and not byte[]. That is me who refactored it to see if it would do any good. Bit shifting don't take many ticks, so I might just revert that.
> I belive the code is quite self explaining.
> InstanciatedIndex ii = ..
> ii.new InstanciatedIndexReader();
> ii.addDocument(s).. replace IndexWriter for now.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org