You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@manifoldcf.apache.org by "Piergiorgio Lucidi (Jira)" <ji...@apache.org> on 2026/05/25 16:23:00 UTC
[jira] [Comment Edited] (CONNECTORS-1784) Core Performance & Modernization (The Java 21 Leap)
[ https://issues.apache.org/jira/browse/CONNECTORS-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083327#comment-18083327 ]
Piergiorgio Lucidi edited comment on CONNECTORS-1784 at 5/25/26 4:22 PM:
-------------------------------------------------------------------------
Here are some of the main tasks currently implemented in the CONNECTORS-1784 branch:
* Migrated the build process to Maven: introduced a new distribution module for generating all the needed resources replicating the Ant build
* Project upgraded to OpenJDK 25 in order to facilitate Virtual Threads and Observability.
* Enabled Preview Features: Updated the Maven build (pom.xml) to include the --enable-preview argument for the maven-compiler-plugin, maven-surefire-plugin, and maven-failsafe-plugin, allowing the use of the StructuredTaskScope API.
* First implementation of Virtual Threads and Structured Concurrency.
* Jetty upgraded to version 12.1.8.
* Enabled JSP Support for Jetty 12 (EE 10).
* asm upgraded to version 9.10.
* Crawler Engine (CrawlerAgent): Replaced fixed-size arrays of platform threads for Worker, Expire, Delete, and Cleanup pools with Virtual Threads started via Thread.ofVirtual().
* Updated StructuredTaskScope usage to the Java 25 API, replacing ShutdownOnFailure with the new StructuredTaskScope.open() and Joiner patterns.
* Refactored the per-query execution mechanism. Previously, ManifoldCF spawned isolated ExecuteQueryThread instances (to bypass issues with drivers swallowing interrupts). These are now executed within a StructuredTaskScope.ShutdownOnFailure(), ensuring that any timeout or interruption properly cleans up the database connection and avoids leaked query threads.
* Refactor for the Authority Service (UserACLServlet): Parallel execution of mapping and authorization checks now leverages Structured Concurrency. If an error occurs while communicating with one authority, the structured scope ensures all parallel constraint matches are safely cancelled and joined, rather than leaving dangling threads in the servlet container.
* Refactor for the Web Crawler Connector (ThrottledFetcher): Replaced the manually managed ExecuteMethodThread with a structured fork() call. This allows the web crawler to reliably abort lingering HTTP connections to non-responsive web servers without leaking resources during complex web snapshots.
* Refactored CmisRepositoryConnector and CmisOutputConnector to execute their session acquisition, connection checks, and teardown routines inside a StructuredTaskScope.
* Updated ModifiedCloudSolrClient to use Structured Concurrency for parallel shard updates, replacing the legacy ThreadPoolExecutor and Future.get() loops. If one replica fails during an update, the scope manages the failure cleanly and prevents orphaned update tasks from consuming resources.
was (Author: piergiorgiolucidi@gmail.com):
Here are some of the main tasks currently implemented in the CONNECTORS-1784 branch:
* Project upgraded to OpenJDK 25 in order to facilitate Virtual Threads and Observability.
* Enabled Preview Features: Updated the Maven build (pom.xml) to include the --enable-preview argument for the maven-compiler-plugin, maven-surefire-plugin, and maven-failsafe-plugin, allowing the use of the StructuredTaskScope API.
* First implementation of Virtual Threads and Structured Concurrency.
* Jetty upgraded to version 12.1.8.
* Enabled JSP Support for Jetty 12 (EE 10).
* asm upgraded to version 9.10.
* Crawler Engine (CrawlerAgent): Replaced fixed-size arrays of platform threads for Worker, Expire, Delete, and Cleanup pools with Virtual Threads started via Thread.ofVirtual().
* Updated StructuredTaskScope usage to the Java 25 API, replacing ShutdownOnFailure with the new StructuredTaskScope.open() and Joiner patterns.
* Refactored the per-query execution mechanism. Previously, ManifoldCF spawned isolated ExecuteQueryThread instances (to bypass issues with drivers swallowing interrupts). These are now executed within a StructuredTaskScope.ShutdownOnFailure(), ensuring that any timeout or interruption properly cleans up the database connection and avoids leaked query threads.
* Refactor for the Authority Service (UserACLServlet): Parallel execution of mapping and authorization checks now leverages Structured Concurrency. If an error occurs while communicating with one authority, the structured scope ensures all parallel constraint matches are safely cancelled and joined, rather than leaving dangling threads in the servlet container.
* Refactor for the Web Crawler Connector (ThrottledFetcher): Replaced the manually managed ExecuteMethodThread with a structured fork() call. This allows the web crawler to reliably abort lingering HTTP connections to non-responsive web servers without leaking resources during complex web snapshots.
* Refactored CmisRepositoryConnector and CmisOutputConnector to execute their session acquisition, connection checks, and teardown routines inside a StructuredTaskScope.
* Updated ModifiedCloudSolrClient to use Structured Concurrency for parallel shard updates, replacing the legacy ThreadPoolExecutor and Future.get() loops. If one replica fails during an update, the scope manages the failure cleanly and prevents orphaned update tasks from consuming resources.
> Core Performance & Modernization (The Java 21 Leap)
> ---------------------------------------------------
>
> Key: CONNECTORS-1784
> URL: https://issues.apache.org/jira/browse/CONNECTORS-1784
> Project: ManifoldCF
> Issue Type: Task
> Components: Framework core
> Reporter: Piergiorgio Lucidi
> Assignee: Piergiorgio Lucidi
> Priority: Major
>
> The transition to OpenJDK 21 is the foundation for a more scalable and responsive architecture.
> *Virtual Threads Integration (Project Loom):*
> * Replace legacy thread pooling with *Virtual Threads* to handle thousands of concurrent repository connections with minimal memory overhead.
> * Implement *Structured Concurrency* to improve the reliability of complex crawling jobs and prevent resource leakage.
> *REST API v2 & OpenAPI Specification:*
> * Complete the transition to a fully documented RESTful API.
> * Enable "Configuration-as-Code" to support modern DevOps workflows and external orchestration.
> *Observability with OpenTelemetry:*
> * Integrate native tracing to monitor document processing latency across connectors, leveraging Java 21's improved profiling capabilities.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)