You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "drempapis (via GitHub)" <gi...@apache.org> on 2026/04/06 14:29:04 UTC
[PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
drempapis opened a new pull request, #15937:
URL: https://github.com/apache/lucene/pull/15937
### Problem
Lucene's `Operations.determinize` uses powerset construction to convert an NFA into a DFA. For patterns with combinatorial structure (e.g. *a*b*c*d*e*f*), this can cause exponential blowup in the number of DFA states, each carrying its own FrozenIntSet snapshot, HashMap entry, and backing arrays. The existing `workLimit` parameter bounds `CPU` effort but provides no memory signal, the JVM can `OOM` long before the work limit is reached. There is currently no way for callers to observe or react to memory growth during determinization.
### Update
This PR adds a new overload to Lucene's determinization API
```
public static Automaton determinize(Automaton a, int workLimit, LongConsumer progressCallback)
```
When a non-null callback is provided, Lucene periodically invokes it with an accumulated estimate of bytes allocated since the last report. The caller can use this signal to enforce memory policies (e.g. throw a circuit-breaker exception to abort determinization)
The estimate is built from the allocations directly attributable to each newly discovered DFA state during powerset construction. Rather than invoking the callback on every new state (which would add overhead proportional to state count), estimates are accumulated and reported in chunks once a configurable byte threshold is crossed. Any remainder is flushed once at the end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "drempapis (via GitHub)" <gi...@apache.org>.
drempapis commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4214759162
Closing this. Further work in #15939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "drempapis (via GitHub)" <gi...@apache.org>.
drempapis commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4196994787
Thanks @rmuir , this is a fair concern.
My intent with the callback was to support callers that already control the determinization call site directly, not to immediately thread a new parameter through WildcardQuery, RegexpQuery, tokenizers, etc.
That said, I agree with your point that for a stable Lucene API, if we introduce this as a public control, we should likely propagate it consistently through higher-level entry points (similar to how `determinizeWorkLimit` is exposed today).
Would you prefer we move in that propagation direction, or do you have a different API shape in mind? I’m happy to adjust accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "drempapis (via GitHub)" <gi...@apache.org>.
drempapis commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4199316552
Thank you @rmuir , this makes sense to me. I agree that adding more resource-control knobs around determinization feels like the wrong direction for Lucene.
I’ll experiment with the alternative path you suggested, removing upfront determinization from WildcardQuery and RegexpQuery, and relying on existing execution-path selection. Does this look good to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4199829902
> I’ll experiment with the alternative path you suggested, removing upfront determinization from WildcardQuery and RegexpQuery, and relying on existing execution-path selection. Does this look good to you?
+1, we should really try to do this. In many cases you'll get a DFA in practice today... even often a minimal one, right from these parsers, unless you are using kleene star, which is going to be slow anyway. So I think what is fast will stay fast and what is slow will stay slow :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "drempapis (via GitHub)" <gi...@apache.org>.
drempapis closed pull request #15937: Add LongConsumer progress callback to Operations.determinize for memo…
URL: https://github.com/apache/lucene/pull/15937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4198284211
> My intent with the callback was to support callers that already control the determinization call site directly, not to immediately thread a new parameter through WildcardQuery, RegexpQuery, tokenizers, etc.
Well, I think lucene shouldn't have even a single parameter for this resource control, and definitely not two parameters. Such parameters don't make sense to me and only come from people who think they can expose exponential algorithms safely to hostile user inputs. You can't: so just don't do that: it isn't safe and it isn't lucene's job to make it safe. Adding more resource controls and making the api even worse, will not change that situation.
I'd like to remove determinization from WildcardQuery/RegexpQuery at some point. If the input is a DFA, then you get a DFA execution, otherwise an NFA execution. The code is already there for this. We just need to flip the switch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add LongConsumer progress callback to Operations.determinize for memo… [lucene]
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on PR #15937:
URL: https://github.com/apache/lucene/pull/15937#issuecomment-4196416386
I'm concerned about two parameters on determinize() to control the resource usage. The PR just adds it in one place, but then its only a matter of time before bugs are filed because the additional parameter isn't exposed in WildcardQuery, RegexpQuery, and then everywhere else (tokenizers, whatever). so I think the API needs to be re-thought.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org