You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Doron Cohen (Created) (JIRA)" <ji...@apache.org> on 2012/02/02 17:22:51 UTC
[jira] [Created] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
--------------------------------------------------------------------------------------
Key: LUCENE-3746
URL: https://issues.apache.org/jira/browse/LUCENE-3746
Project: Lucene - Java
Issue Type: Bug
Components: modules/spellchecker
Reporter: Doron Cohen
Fix For: 3.6, 4.0
Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-3746:
--------------------------------
Attachment: LUCENE-3746.patch
Simple fix: consult also with maxMemory if freeMemory not suffice.
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-3746:
--------------------------------
Attachment: LUCENE-3746.patch
Updated patch - without MemoryMXBean - computing 'max, total, free' (in that order) and deciding by 'free' or falling to 'max-free'. This is more conservative, than MemoryMxBean but since the latter is not full proof either, I prefer the simpler approach.
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200834#comment-13200834 ]
Dawid Weiss commented on LUCENE-3746:
-------------------------------------
As for spawning MemoryPoolMXBean -- I wouldn't be worried about it, it's probably a system daemon thread for sending memory threshold notifications (didn't check though). I will peek at openjdk sources and see how the mx is implemented to verify if it's atomic or not (not a guarantee, just curiosity).
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201073#comment-13201073 ]
Doron Cohen commented on LUCENE-3746:
-------------------------------------
Thanks Dawid!
{quote}
it's probably a system daemon thread for sending memory threshold notifications
{quote}
Yes this makes sense.
Still the difference between the two JDKs felt bothering.
Some more digging, and now I think it is clear.
Here are the stack traces reported (at the end of the test) with Oracle:
{noformat}
1. Thread[ReaderThread,5,main]
2. Thread[main,5,main]
3. Thread[Reference Handler,10,system]
4. Thread[Signal Dispatcher,9,system]
5. Thread[Finalizer,8,system]
6. Thread[Attach Listener,5,system]
{noformat}
And with IBM JDK:
{noformat}
1. Thread[Attach API wait loop,10,main]
2. Thread[Finalizer thread,5,system]
3. Thread[JIT Compilation Thread,10,system]
4. Thread[main,5,main]
5. Thread[Gc Slave Thread,5,system]
6. Thread[ReaderThread,5,main]
7. Thread[Signal Dispatcher,5,main]
8. Thread[MemoryPoolMXBean notification dispatcher,6,main]
{noformat}
The 8th thread is the one that started only after accessing the memory management layer. The thing is, that in the IBM JDK that thread is created in the ThreadGroup "main", while in the Oracle JDK it is created under "system". To me the latter makes more sense.
To be more sure I added a fake memory notification listener and check the thread in which notification happens:
{code}
MemoryMXBean mmxb = ManagementFactory.getMemoryMXBean();
NotificationListener listener = new NotificationListener() {
@Override
public void handleNotification(Notification notification, Object handback) {
System.out.println(Thread.currentThread());
}
};
((NotificationEmitter) mmxb).addNotificationListener(listener, null, null);
{code}
Evidently in IBM JDK the notification is in "main" group thread (also in line with the thread-group in the original warning message which triggered this threads discussion):
{noformat}
Thread[MemoryPoolMXBean notification dispatcher,6,main]
{noformat}
While in Oracle JDK notification is in "system" group thread:
{noformat}
Thread[Low Memory Detector,9,system]
{noformat}
This also explains why the warning is reported only for IBM JDK: because the threads check in LTC only account for the threads in the same thread-group as the one running the specific test case. So when dispatching happens in a "system" group thread it is not sensed by that check at all.
Ok now with mystery solved I can commit the simpler code...
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Updated] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen updated LUCENE-3746:
--------------------------------
Attachment: LUCENE-3746.patch
Updated patch using ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().
Javadocs are not explicit about this call being atomic, but from the wording it seems almost certain to conclude that each call returns a new Usage instance. In this patch this is (Java) asserted and the assert passes (-ea) in two different JVMs - IBM and Oracle - so this might be correct. I searched some more explicit info on this with no success.
Annoyingly though, in IBM JDK, running the tests like this produces the nice warning:
{noformat}
WARNING: test class left thread running: Thread[MemoryPoolMXBean notification dispatcher,6,main]
RESOURCE LEAK: test class left 1 thread(s) running
{noformat}
This makes me reluctant to use the memory bean - I did not find a way to prevent that thread leak.
So perhaps a better approach would be to be conservative about the sequence of calls when using Runtime? something like this:
{code}
long free = rt.freeMemory();
if (free is sufficient)
return decideBy(free);
long max = rt.maxMemory();
long total = rt.totalMemory();
return decideBy(max - total)
{code}
This is conservative in that 'total' is computed last, and in that total-free is not added to the computed available bytes.
In both approaches, even if atomicity is guaranteed, it is possible that more heap is allocated in another thread between the time that the size is computed, to the time that the bytes are actually allocated, so not sure how safe this check can be made.
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Dawid Weiss (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200843#comment-13200843 ]
Dawid Weiss commented on LUCENE-3746:
-------------------------------------
Just checked and it seems to be that within a single memory pool the results will be atomic. Unfortunately that call aggregates all memory pools and (depending on the GC used) this may result in inconsistencies if the calculation happens to be interwoven with garbage collector activity. As stated in the sources of G1, for example:
{noformat}
// 4) Now, there is a very subtle issue with all the above. The
// framework will call get_memory_usage() on the three pools
// asynchronously. As a result, each call might get a different value
// for, say, survivor_num which will yield inconsistent values for
// eden_used, survivor_used, and old_gen_used (as survivor_num is used
// in the calculation of all three). This would normally be
// ok. However, it's possible that this might cause the sum of
// eden_used, survivor_used, and old_gen_used to go over the max heap
// size and this seems to sometimes cause JConsole (and maybe other
// clients) to get confused. There's not a really an easy / clean
// solution to this problem, due to the asynchrounous nature of the
// framework.
{noformat}
Makes sense to me. I wouldn't bother with management interface then and just use the Runtime.* heuristic you proposed.
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Resolved] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doron Cohen resolved LUCENE-3746.
---------------------------------
Resolution: Fixed
Assignee: Doron Cohen
Lucene Fields: Patch Available (was: New)
Committed:
- r1241355 - trunk
- r1241363 - 3x
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Assignee: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch, LUCENE-3746.patch, LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] [Commented] (LUCENE-3746) suggest.fst.Sort.BufferSize should
not automatically fail just because of freeMemory()
Posted by "Doron Cohen (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13199038#comment-13199038 ]
Doron Cohen commented on LUCENE-3746:
-------------------------------------
{quote}
[Dawid:|http://markmail.org/message/jobtemqm4u4vrxze] (maxMemory - totalMemory) because that's how much the heap can
grow? The problem is none of this is atomic, so the result can
unpredictable. There are other methods in management interface that
permit a somewhat more detailed checks. Don't know if they guarantee
atomicity of the returned snapshot, but I doubt it.
- [MemoryMXBean.getHeapMemoryUsage()|http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryMXBean.html#getHeapMemoryUsage()]
- [MemoryPoolMXBean.getPeakUsage()|http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/management/MemoryPoolMXBean.html#getPeakUsage()]
{quote}
Current patch not (yet) handling the atomicity issue Dawid described.
> suggest.fst.Sort.BufferSize should not automatically fail just because of freeMemory()
> --------------------------------------------------------------------------------------
>
> Key: LUCENE-3746
> URL: https://issues.apache.org/jira/browse/LUCENE-3746
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/spellchecker
> Reporter: Doron Cohen
> Fix For: 3.6, 4.0
>
> Attachments: LUCENE-3746.patch
>
>
> Follow up op dev thread: [FSTCompletionTest failure "At least 0.5MB RAM buffer is needed" | http://markmail.org/message/d7ugfo5xof4h5jeh]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org