You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Hoss Man (JIRA)" <ji...@apache.org> on 2009/08/10 20:37:17 UTC
[jira] Created: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
FieldCacheSanityChecker called directly by FieldCache.get*
----------------------------------------------------------
Key: LUCENE-1798
URL: https://issues.apache.org/jira/browse/LUCENE-1798
Project: Lucene - Java
Issue Type: Improvement
Components: Search
Reporter: Hoss Man
As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747564#action_12747564 ]
Michael McCandless commented on LUCENE-1798:
--------------------------------------------
I think that may be over-designing things...
Ie, I'm thinking by far the most common use here is a user who can't
understand why their FieldCache memory has increased so much, and,
they want a simple & fast way to debug it.
If we switch to the InsanityMonitor API then they'll have to implement
a class w/ that interface and a chunk of code that does the for loop,
printing out details?
Is there any other use case besides "help me debug" that would merit a
programmatic API?
I suppose we could switch to InsanityMonitor but then provide a
PrintStreamInstanityMonitor impl... still seems kinda overkill though.
But I don't feel strongly. Do you want to take a crack at it?
I think what's important here is there's a one-line means for a user
who's having problems to "turn on FieldCache debugging" and see
juicy details when insanity increases.
I agree we could make LuceneTestCase.tearDown more robust if tap into
this, though the simple infoStream could also be used for that?
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch, LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747538#action_12747538 ]
Hoss Man commented on LUCENE-1798:
----------------------------------
Michael: reading the patch you commited, i only have two concerns...
1) a PrintStream doesn't really seem like the ideal callback API for this situation ... with IndexWriter it makes some sense because we ant to be able to log all sorts of misc info that will be unstructured, but in the field cache checkign case we already have a fairly robust data structure (Insanity) that we can provide ... so instead of a setInfoStream(PritStream) method, why not have a callback interface that takes Insanity objects (and the Entry that triggered the problem)
{code}
/**
* If non-null, this monitor will be notify anytime an entry is created
* which are not sane according to {@link FieldCacheSanityChecker}.
* @param monitor The Monitor to notify, if it throws a RuntimeException then the cache method will throw a RuntimeException.
*/
public void setInsanityMonitor(InsanityMonitor monitor)
...
public interface InsanityMonitor {
public void notify(CacheEntry e, Instanty[] i);
}
{code}
2) it seems like we should change LuceneTestCase to use this new hook instead of just calling the FieldCacheSanityChecker in tearDown() ... that way we can be sure we're checking all FieldCache usages (the current approach risks IndexReader weak refs getting gc'ed after they go out of scope in the test and before the checker runs in tearDown)
....thoughts?
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch, LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747570#action_12747570 ]
Hoss Man commented on LUCENE-1798:
----------------------------------
bq. I suppose we could switch to InsanityMonitor but then provide a PrintStreamInstanityMonitor impl... still seems kinda overkill though.
...that was what i had in mind, but you're right -- it is overkill. a PrintStream is a nice quick and easy way to get this info -- if they really want robust data structures they can use the sanity checker directly (possibly even from a mock PrintStream)
bq. I agree we could make LuceneTestCase.tearDown more robust if tap into this, though the simple infoStream could also be used for that? ... sure, because if *anything* gets written to that stream, it indicates a bug ... unless they expect it, in which case they can catch an exception an ignore it.
but the LuceneTestCase changes are less urgent ... i was mainly worried about making surewe were happy with the API. You've convinced me.
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch, LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1798:
---------------------------------------
Fix Version/s: 2.9
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1798:
---------------------------------------
Attachment: LUCENE-1798.patch
Attached patch. I added get/setInfoStream to FieldCache, then, in FieldCacheImpl.Cache.get, if we hit a cache miss and infoStream is enabled, I gather the Insanity[] before the cache entry is added and after, then print out any change involving the entry just added. It produces this output to the infoStream:
{noformat}
[junit] WARNING: new FieldCache insanity created
[junit] Details: VALUEMISMATCH: Multiple distinct value objects for org.apache.lucene.index.DirectoryReader@da3a1e+theDouble
[junit] 'org.apache.lucene.index.DirectoryReader@da3a1e'=>'theDouble',float,org.apache.lucene.search.FieldCache.DEFAULT_FLOAT_PARSER=>[F#7896426 (size =~ 3.9 KB)
[junit] 'org.apache.lucene.index.DirectoryReader@da3a1e'=>'theDouble',double,org.apache.lucene.search.FieldCache.DEFAULT_DOUBLE_PARSER=>[D#5503831 (size =~ 7.8 KB)
[junit] 'org.apache.lucene.index.DirectoryReader@da3a1e'=>'theDouble',double,null=>[D#5503831 (size =~ 7.8 KB)
[junit]
[junit]
[junit] Stack:
[junit]
[junit] java.lang.Throwable
[junit] at org.apache.lucene.search.FieldCacheImpl$Cache.printNewInsanity(FieldCacheImpl.java:263)
[junit] at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:228)
[junit] at org.apache.lucene.search.FieldCacheImpl.getFloats(FieldCacheImpl.java:494)
[junit] at org.apache.lucene.search.FieldCacheImpl$FloatCache.createValue(FieldCacheImpl.java:509)
[junit] at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:223)
[junit] at org.apache.lucene.search.FieldCacheImpl.getFloats(FieldCacheImpl.java:494)
[junit] at org.apache.lucene.search.FieldCacheImpl.getFloats(FieldCacheImpl.java:487)
[junit] at org.apache.lucene.search.TestFieldCache.testInfoStream(TestFieldCache.java:70)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[junit] at java.lang.reflect.Method.invoke(Method.java:597)
[junit] at junit.framework.TestCase.runTest(TestCase.java:164)
[junit] at junit.framework.TestCase.runBare(TestCase.java:130)
[junit] at org.apache.lucene.util.LuceneTestCase.runBare(LuceneTestCase.java:206)
[junit] at junit.framework.TestResult$1.protect(TestResult.java:106)
[junit] at junit.framework.TestResult.runProtected(TestResult.java:124)
[junit] at junit.framework.TestResult.run(TestResult.java:109)
[junit] at junit.framework.TestCase.run(TestCase.java:120)
[junit] at junit.framework.TestSuite.runTest(TestSuite.java:230)
[junit] at junit.framework.TestSuite.run(TestSuite.java:225)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
[junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
{noformat}
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746783#action_12746783 ]
Michael McCandless commented on LUCENE-1798:
--------------------------------------------
bq. Why not just a single call after the val has been created and log if any of the Insanity objects contain the new val?
I was worried about over-printing of previously created insanities, but you're right: since the new entry was just created, there's no way any insanity involving this entry would have been printed before (duh!). So I'll simplify it...
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12741480#action_12741480 ]
Hoss Man commented on LUCENE-1798:
----------------------------------
https://issues.apache.org/jira/browse/LUCENE-1749?focusedCommentId=12741479#action_12741479
{quote}
FieldCacheImpl.Cache.get could use the FieldCacheSanityChecker to inspect itself immediately after calling createValue, and could even test if any of the Insanity instances returned are related to the current call (by comparing the CacheEntry with the Entry it's using) ... it could even log a useful stack trace since the sanity check would be happening in the same call stack as at least one of the CacheEntries in the Insanity object.
{quote}
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-1798.
----------------------------------------
Resolution: Fixed
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch, LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12746719#action_12746719 ]
Hoss Man commented on LUCENE-1798:
----------------------------------
i haven't looked at the patch, but i don't think you need two calls to the sanity checker.
Why not just a single call after the val has been created and log if any of the Insanity objects contain the new val?
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless reassigned LUCENE-1798:
------------------------------------------
Assignee: Michael McCandless
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-1798) FieldCacheSanityChecker called
directly by FieldCache.get*
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-1798:
---------------------------------------
Attachment: LUCENE-1798.patch
New patch attached.
> FieldCacheSanityChecker called directly by FieldCache.get*
> ----------------------------------------------------------
>
> Key: LUCENE-1798
> URL: https://issues.apache.org/jira/browse/LUCENE-1798
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Search
> Reporter: Hoss Man
> Assignee: Michael McCandless
> Fix For: 2.9
>
> Attachments: LUCENE-1798.patch, LUCENE-1798.patch
>
>
> As suggested by McCandless in LUCENE-1749, we can make FieldCacheImpl a client of the FieldCacheSanityChecker and have it sanity check itself each time it creates a new cache entry, and log a warning if it thinks there is a problem. (although we'd probably only want to do this if the caller has set some sort of infoStream/warningStream type property on the FieldCache object.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org