You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by "allthingssecurity (via GitHub)" <gi...@apache.org> on 2026/10/10 00:04:41 UTC
[PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
allthingssecurity opened a new pull request, #27639:
URL: https://github.com/apache/camel/pull/27639
# Description
[CAMEL-25506](https://issues.apache.org/jira/browse/CAMEL-25506)
`LuceneQueryProducer` used one `LuceneSearcher` for all exchanges, and `LuceneSearcher` kept the reader, the `IndexSearcher` and the hits of the last query in fields. With concurrent exchanges one exchange could get the hits of another exchange's query (27 to 48 of 20000 queries in the new test on main), and every query's reader except the last one was left to the garbage collector. `doStop()` closed the endpoint's analyzer, so after a route restart every query failed with `AlreadyClosedException: this Analyzer is closed`; a stop before any query threw a `NullPointerException`.
This change uses a `LuceneSearcher` per exchange (open, search, close in a `finally`) in `LuceneQueryProducer` and `LuceneQueryProcessor`, and `LuceneSearcher.close()` closes only its reader (null-safe), not the analyzer, which belongs to the endpoint or the registry. Code that uses `LuceneSearcher` directly must now close its analyzer itself. The `LuceneQueryProcessor` change is the same idiom and is covered only by the existing `LuceneQueryProcessorIT`.
This is one of three independent camel-lucene fixes of the same batch (inserts after a restart, query producer state, per-endpoint configuration); they change different files and apply in any order.
Tests:
- New `LuceneQueryProducerLifecycleTest` in camel-lucene: a query after a route restart (deterministic), and 20000 concurrent queries for two terms on 8 threads, each checked against its own term (about 2 s; it cannot force the interleaving, it makes it very likely).
- Without the change: `a query after the route was restarted must succeed ==> expected: <null> but was: <org.apache.lucene.store.AlreadyClosedException: this Analyzer is closed>` and `each query must get the hits of its own query, but 39 of 20000 did not` (both tests fail).
- With the change, camel-lucene tests pass: `LuceneQueryProducerLifecycleTest` (2), `LuceneIndexAndQueryProducerIT` (4), `LuceneQueryProcessorIT` (2), 0 failures.
Found with a TLA+ model of the producer and searcher (one action per field read/write, route stop/start): "every answer holds the hits of its own query" is violated in 7 steps with two threads, and "no query runs on a closed analyzer" in 9 steps (query, stop, start, query). I then reproduced it with the real component.
# Target
- [x] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch)
# Tracking
- [x] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it).
# Apache Camel coding standards and style
- [x] I checked that each commit in the pull request has a meaningful subject line and body.
- [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes.
(I built and tested camel-lucene, including the formatter and import-sort plugins. I did not run the full root build.)
# AI-assisted contributions
- [x] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used.
This PR was prepared with Claude Code (Claude Opus 5.5). The commit carries a `Co-Authored-By` trailer.
_Claude Code on behalf of allthingssecurity_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on code in PR #27639:
URL: https://github.com/apache/camel/pull/27639#discussion_r4236604261
##########
components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneSearcher.java:
##########
@@ -52,9 +55,15 @@ public void open(File indexDirectory, Analyzer analyzer) throws IOException {
this.analyzer = analyzer;
}
+ /**
+ * Closes the index reader. The analyzer is not closed: it belongs to the endpoint (or the registry) and is used by
+ * the next query.
+ */
public void close() throws IOException {
Review Comment:
`LuceneSearcher` is a public class, and `close()` no longer closes the analyzer passed to `open()`. As the PR description says, anyone using it directly now has to close the analyzer themselves. Could you add a one-line note under a `camel-lucene` heading in `docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_23.adoc`? Non-blocking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27639:
URL: https://github.com/apache/camel/pull/27639#issuecomment-6094586793
<!-- ci-tested-modules -->
:test_tube: **CI tested the following changed modules:**
- `components/camel-lucene`
---
<details><summary>:microscope: Scalpel shadow comparison — Scalpel: 9 of 704 tested, 27 compile-only — current: 9 all tested</summary>
[Maveniverse Scalpel](https://github.com/maveniverse/scalpel) detected **9 affected modules** (current approach: 9).
**Skip-tests mode would test 9 modules** (1 direct + 8 downstream), **skip tests for 27** (generated code, meta-modules)
<details><summary>Modules Scalpel would test (9)</summary>
- `camel-jbang-mcp` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-mcp` ← downstream of org.apache.camel:camel-jbang-core
- `camel-jbang-plugin-route-parser` ← downstream of org.apache.camel:camel-route-parser
- `camel-jbang-plugin-tui` ← downstream of org.apache.camel:camel-catalog
- `camel-jbang-plugin-validate` ← downstream of org.apache.camel:camel-yaml-dsl-validator
- `camel-launcher-container` ← downstream of org.apache.camel:camel-launcher
- `camel-lucene` ← components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneQueryProducer.java, components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneSearcher.java, components/camel-lucene/src/main/java/org/apache/camel/processor/lucene/LuceneQueryProcessor.java, components/camel-lucene/src/test/java/org/apache/camel/component/lucene/LuceneQueryProducerLifecycleTest.java
- `camel-yaml-dsl-validator` ← downstream of org.apache.camel:camel-catalog
- `camel-yaml-dsl-validator-maven-plugin` ← downstream of org.apache.camel:camel-yaml-dsl-validator
</details>
<details><summary>Modules with tests skipped (27)</summary>
- `apache-camel`
- `camel-allcomponents`
- `camel-catalog`
- `camel-catalog-console`
- `camel-catalog-maven`
- `camel-catalog-suggest`
- `camel-componentdsl`
- `camel-endpointdsl`
- `camel-endpointdsl-support`
- `camel-itest`
- `camel-jbang-core`
- `camel-jbang-it`
- `camel-jbang-main`
- `camel-jbang-plugin-edit`
- `camel-jbang-plugin-generate`
- `camel-jbang-plugin-kubernetes`
- `camel-jbang-plugin-test`
- `camel-kamelet-main`
- `camel-launcher`
- `camel-report-maven-plugin`
- `camel-route-parser`
- `camel-yaml-dsl`
- `camel-yaml-dsl-deserializers`
- `camel-yaml-dsl-maven-plugin`
- `coverage`
- `docs`
- `dummy-component`
</details>
> :information_source: Shadow mode — Scalpel observes but does not affect test execution. [Learn more](https://github.com/maveniverse/scalpel)
</details>
<details><summary>All tested modules (36 modules, 6m 38s total)</summary>
**Total reactor time:** 6m 38s
| Module | Duration | Status |
| --- | --- | --- |
| Camel :: JBang :: Plugin :: TUI | 55.0s | SUCCESS |
| Camel :: Launcher | 52.4s | SUCCESS |
| Camel :: JBang :: MCP | 42.1s | SUCCESS |
| Camel :: Component DSL | 32.2s | SUCCESS |
| Camel :: YAML DSL :: Validator | 23.4s | SUCCESS |
| Camel :: Catalog :: Camel Catalog | 23.4s | SUCCESS |
| Camel :: JBang :: Plugin :: Kubernetes | 20.4s | SUCCESS |
| Camel :: YAML DSL | 20.3s | SUCCESS |
| Camel :: JBang :: Plugin :: Validate | 17.6s | SUCCESS |
| Camel :: JBang :: Plugin :: Testing | 14.1s | SUCCESS |
| Camel :: Lucene | 14.0s | SUCCESS |
| Camel :: Docs | 12.3s | SUCCESS |
| Camel :: Kamelet Main | 11.3s | SUCCESS |
| Camel :: Catalog :: Camel Route Parser | 8.5s | SUCCESS |
| Camel :: YAML DSL :: Deserializers | 8.5s | SUCCESS |
| Camel :: Catalog :: Camel Report Maven Plugin | 6.8s | SUCCESS |
| Camel :: All Components Sync point | 6.0s | SUCCESS |
| Camel :: YAML DSL :: Validator Maven Plugin | 5.1s | SUCCESS |
| Camel :: Catalog :: Maven | 3.9s | SUCCESS |
| Camel :: YAML DSL :: Maven Plugins | 3.5s | SUCCESS |
| Camel :: Catalog :: Suggest (deprecated) | 2.7s | SUCCESS |
| Camel :: Assembly | 2.3s | SUCCESS |
| Camel :: JBang :: Plugin :: Edit | 2.1s | SUCCESS |
| Camel :: Coverage | 1.7s | SUCCESS |
| Camel :: JBang :: Plugin :: Generate | 1.5s | SUCCESS |
| Camel :: JBang :: Plugin :: MCP | 1.3s | SUCCESS |
| Camel :: Catalog :: Dummy Component | 1.2s | SUCCESS |
| Camel :: JBang :: Main | 0.9s | SUCCESS |
| Camel :: JBang :: Integration tests | 0.8s | SUCCESS |
| Camel :: Endpoint DSL :: Support | 0.8s | SUCCESS |
| Camel :: Catalog :: Console | 0.8s | SUCCESS |
| Camel :: Launcher :: Container | 0.7s | SUCCESS |
| Camel :: JBang :: Plugin :: Route Parser | 0.7s | SUCCESS |
| Camel :: Endpoint DSL | n/a | |
| Camel :: Integration Tests | n/a | |
| Camel :: JBang :: Core | n/a | |
**Top 20 slowest modules:**
- `Camel :: JBang :: Plugin :: TUI` (55.0s)
- `Camel :: Launcher` (52.4s)
- `Camel :: JBang :: MCP` (42.1s)
- `Camel :: Component DSL` (32.2s)
- `Camel :: YAML DSL :: Validator` (23.4s)
- `Camel :: Catalog :: Camel Catalog` (23.4s)
- `Camel :: JBang :: Plugin :: Kubernetes` (20.4s)
- `Camel :: YAML DSL` (20.3s)
- `Camel :: JBang :: Plugin :: Validate` (17.6s)
- `Camel :: JBang :: Plugin :: Testing` (14.1s)
- `Camel :: Lucene` (14.0s)
- `Camel :: Docs` (12.3s)
- `Camel :: Kamelet Main` (11.3s)
- `Camel :: Catalog :: Camel Route Parser` (8.5s)
- `Camel :: YAML DSL :: Deserializers` (8.5s)
- `Camel :: Catalog :: Camel Report Maven Plugin` (6.8s)
- `Camel :: All Components Sync point` (6.0s)
- `Camel :: YAML DSL :: Validator Maven Plugin` (5.1s)
- `Camel :: Catalog :: Maven` (3.9s)
- `Camel :: YAML DSL :: Maven Plugins` (3.5s)
</details>
---
:gear: [View full build and test results](https://github.com/apache/camel/actions/runs/38007447867)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27639:
URL: https://github.com/apache/camel/pull/27639#issuecomment-6094368681
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:robot: CI automation will test this PR automatically.
:camel: Apache Camel Committers, please review the following items:
* First-time contributors **require MANUAL approval** for the GitHub Actions to run
* You can use the command `/component-test (camel-)component-name1 (camel-)component-name2..` to request a test from the test bot although they are normally detected and executed by CI.
* You can label PRs using `skip-tests` and `test-dependents` to fine-tune the checks executed by this PR.
* Build and test logs are available in the summary page. **Only** [Apache Camel committers](https://camel.apache.org/community/team/#committers) have access to the summary.
:warning: Be careful when sharing logs. Review their contents before sharing them publicly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus merged PR #27639:
URL: https://github.com/apache/camel/pull/27639
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] CAMEL-25506: camel-lucene - Query with a searcher per exchange and do not close the analyzer on stop [camel]
Posted by "allthingssecurity (via GitHub)" <gi...@apache.org>.
allthingssecurity commented on code in PR #27639:
URL: https://github.com/apache/camel/pull/27639#discussion_r4236654068
##########
components/camel-lucene/src/main/java/org/apache/camel/component/lucene/LuceneSearcher.java:
##########
@@ -52,9 +55,15 @@ public void open(File indexDirectory, Analyzer analyzer) throws IOException {
this.analyzer = analyzer;
}
+ /**
+ * Closes the index reader. The analyzer is not closed: it belongs to the endpoint (or the registry) and is used by
+ * the next query.
+ */
public void close() throws IOException {
Review Comment:
Added in de8a097f10: a `=== camel-lucene` section in the 4.23 upgrade guide saying that `LuceneSearcher.close()` no longer closes the analyzer passed to `open()`. It still merges cleanly with #27638 and #27640.
_Claude Code on behalf of allthingssecurity_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org