You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Simon Willnauer (JIRA)" <ji...@apache.org> on 2018/04/23 14:20:00 UTC
[jira] [Comment Edited] (LUCENE-8267) Remove memory codecs from the
codebase
[ https://issues.apache.org/jira/browse/LUCENE-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16448208#comment-16448208 ]
Simon Willnauer edited comment on LUCENE-8267 at 4/23/18 2:19 PM:
------------------------------------------------------------------
{quote}If we are going to make it harder to remove stuff, I have no problem being the one to make it equally harder to add stuff.
{quote}
I agree this is one of these issues that we have to face. if we put the bar very high to remove stuff that is not mainstream then we will have a super hard time adding stuff. It creates fear driven decisions. It sucks I agree with [~rcmuir] 100% here.
{quote}
-1 sorry. I've used the MemoryPostingsFormat for a text-tagging use-case where there are intense lookups against the terms dictionary. It's highly beneficial to have the terms dictionary be entirely memory resident, albeit in a compact FST. The issue description mentions "We don't use those memory codecs anywhere outside of tests" – this should be no surprise as it's not the default codec. I'm sure it may be hard to gauge the level of use of something outside of core-Lucene. When we ponder removing something that Lucene doesn't even _need_, I propose we raise the issue more openly to the community. Perhaps the question could be proposed in CHANGES.txt and/or release announcements to solicit community input?
{quote}
given that you know that you are using your veto here we are already in a terrible position to have any conversation. Can you quantify the "it's nice"? since there are alternatives that (standard codec) can you go and provide some numbers. We should not use vetos based on non-quantifiable arguments IMO. We can go and ask the community but I don't expect much useful outcome, most of the folks don't know what they are using here and there. Nevertheless, I am happy to send a mail to dev to get this information.
was (Author: simonw):
{quote}
If we are going to make it harder to remove stuff, I have no problem being the one to make it equally harder to add stuff.
\{quote}
I agree this is one of these issues that we have to face. if we put the bar very high to remove stuff that is not mainstream then we will have a super hard time adding stuff. It creates fear driven decisions. It sucks I agree with [~rcmuir] 100% here.
{quote}
-1 sorry. I've used the MemoryPostingsFormat for a text-tagging use-case where there are intense lookups against the terms dictionary. It's highly beneficial to have the terms dictionary be entirely memory resident, albeit in a compact FST. The issue description mentions "We don't use those memory codecs anywhere outside of tests" – this should be no surprise as it's not the default codec. I'm sure it may be hard to gauge the level of use of something outside of core-Lucene. When we ponder removing something that Lucene doesn't even _need_, I propose we raise the issue more openly to the community. Perhaps the question could be proposed in CHANGES.txt and/or release announcements to solicit community input?
{quote}
given that you know that you are using your veto here we are already in a terrible position to have any conversation. Can you quantify the "it's nice"? since there are alternatives that (standard codec) can you go and provide some numbers. We should not use vetos based on non-quantifiable arguments IMO. We can go and ask the community but I don't expect much useful outcome, most of the folks don't know what they are using here and there. Nevertheless, I am happy to send a mail to dev to get this information.
> Remove memory codecs from the codebase
> --------------------------------------
>
> Key: LUCENE-8267
> URL: https://issues.apache.org/jira/browse/LUCENE-8267
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Dawid Weiss
> Priority: Major
>
> Memory codecs (MemoryPostings*, MemoryDocValues*) are part of random selection of codecs for tests and cause occasional OOMs when a test with huge data is selected. We don't use those memory codecs anywhere outside of tests, it has been suggested to just remove them to avoid maintenance costs and OOMs in tests. [1]
> [1] https://apache.markmail.org/thread/mj53os2ekyldsoy3
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org