[jira] [Commented] (TIKA-4547) Update tika pipes so that it can be properly clustered

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TIKA-4547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18043601#comment-18043601 ] 

ASF GitHub Bot commented on TIKA-4547:
--------------------------------------

nddipiazza opened a new pull request, #2424:
URL: https://github.com/apache/tika/pull/2424

   TIKA-4547 - 
   
   **Phase 1: Foundation (TIKA-4547)**
   - Implement StateStore abstraction
   - Refactor ExpiringFetcherStore
   - Add InMemoryStateStore
   - Update documentation
   




> Update tika pipes so that it can be properly clustered
> ------------------------------------------------------
>
>                 Key: TIKA-4547
>                 URL: https://issues.apache.org/jira/browse/TIKA-4547
>             Project: Tika
>          Issue Type: Task
>            Reporter: Nicholas DiPiazza
>            Priority: Major
>
> Plan: Enable Distributed State Management for Tika Pipes Clustering
> The current Tika Pipes architecture stores Fetcher/Emitter/PipesIterator configurations in local memory (ExpiringFetcherStore using synchronized HashMaps), making it impossible to create a fetcher on one server and use it on another. This plan introduces a pluggable distributed state abstraction to enable true clustering for both gRPC and REST servers.
> * Create StateStore abstraction in tika-pipes-api as an interface with methods put(String key, byte[] value), get(String key), delete(String key), list(), and lifecycle operations, allowing pluggable implementations (in-memory, Apache Ignite, Redis, Hazelcast, etc.).
> * Refactor ExpiringFetcherStore to use StateStore in TikaGrpcServerImpl.java, replacing Collections.synchronizedMap with StateStore API calls for fetchers, fetcherConfigs, and fetcherLastAccessed maps to enable cross-server state sharing.
> * Create parallel EmitterStore and PipesIteratorStore abstractions mirroring ExpiringFetcherStore pattern in tika-pipes-core, applying the same StateStore-backed approach for Emitters and PipesIterators to achieve full component distribution.
> * Add StateStoreFactory plugin system in tika-pipes-core using PF4J pattern (similar to FetcherManager and EmitterManager), loading implementations from Tika config's stateStore section with default in-memory implementation.
> * Update PipesConfig to include state store configuration in PipesConfig.java with fields like stateStoreClass and stateStoreParams, ensuring backward compatibility with local-only deployments via sensible defaults.
> Make PipesClient and PipesServer state-aware by injecting StateStore references in PipesClient.java and PipesServer.java, enabling forked processes to retrieve fetcher/emitter configs from distributed store rather than requiring XML rewrites.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)