[jira] [Resolved] (CAMEL-24621) camel-langchain4j-ingest - document ingestion component: the missing ingestion half of RAG in Camel

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-24621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-24621.
---------------------------------
    Resolution: Fixed

Merged via https://github.com/apache/camel/pull/26190 (commit 4cb6973d018eecd3e90ee129ab55db19783134df).

_Claude Code on behalf of davsclaus_

> camel-langchain4j-ingest - document ingestion component: the missing ingestion half of RAG in Camel
> ---------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-24621
>                 URL: https://issues.apache.org/jira/browse/CAMEL-24621
>             Project: Camel
>          Issue Type: New Feature
>          Components: camel-ai
>            Reporter: Jiří Ondrušek
>            Assignee: Jiri Ondrusek
>            Priority: Major
>             Fix For: 4.23.0
>
>
> Camel today has strong building blocks for the *retrieval* side of RAG — camel-langchain4j-chat and camel-langchain4j-agent consume a vector store, camel-langchain4j-embeddingstore searches one — but nothing that packages getting documents *into* the store. A user who wants a knowledge base has to hand-orchestrate splitting, embedding, batching, metadata and deduplication in every route. [design/langchain4j-evolution.adoc, "Phase 4: The RAG and AIService gap"|https://github.com/apache/camel/blob/main/design/langchain4j-evolution.adoc#phase-4-the-rag-and-aiservice-gap] names this gap explicitly ("there was still no way to do RAG end-to-end"). This ticket proposes closing it with a new component in components/camel-ai.
> *What it brings*
> One line turns any of Camel's 300+ consumers into a RAG ingestion source:
> {code}
> from("aws2-s3://product-docs?deleteAfterRead=false")
>   .to("langchain4j-ingest:products?documentIdHeader=CamelAwsS3Key");
> {code}
> The producer splits the payload into overlapping segments (recursive splitter, configurable sizes), embeds them in batches of 32 (staying under embedding providers' per-request limits), and writes them to any LangChain4j EmbeddingStore — Qdrant, Milvus, pgvector, Weaviate, Neo4j and the rest — stamping every segment with camel_ingest_pipeline and camel_ingest_document_id metadata so retrieval can cite its sources and a future synchronising engine can find a document's vectors again. Data is written the way LangChain4j itself writes it (the same compatibility argument that shaped camel-langchain4j-embeddingstore), so existing retrieval — Camel or plain LangChain4j — finds it.
> On top of the endpoint sit two thin layers, so every audience gets its natural entry point: a declarative Java API (IngestPipelineDefinition + IngestPipelineRouteBuilder — "watch this directory, parse with Tika, dedupe, store there" without writing route internals), and kamelets (langchain4j-ingest-sink, tika/docling variants, a file source) giving camel-jbang, camel-k and Pipe users a zero-route-code experience.
> *Why users will like it*
> * *Nothing new to learn*: it is an ordinary producer endpoint. Store and model beans autowire when the registry holds exactly one of each; with zero or several candidates the endpoint fails fast with the fix spelled out in the message ("name the one to use with embeddingStore=#bean:...").
> * *Continuous by nature*: unlike one-shot loaders (e.g. Easy RAG's startup-only file loading), Camel sources poll and stream — the knowledge base stays current as S3 buckets, Kafka topics or folders change.
> * *Deduplication out of the box*: an optional IdempotentRepository makes redeliveries and re-listings answer "skipped" instead of duplicating vectors — first write wins per document id, a blank or failed delivery releases its claim, and a shared JDBC register gives correct behaviour across replicas.
> * *Safe defaults*: a knowledge base reads its source, it never consumes it (noop, no deletes), half-copied files are waited for, and the document id is captured before any parser runs, so a crafted document cannot forge its own identity.
> * *Honest result contract*: the reply body is an IngestResult (ingested / empty / skipped + segments written), so request-reply callers can react.
> *Provenance and proof*
> The design is not speculative: it is the field-proven engine of the camel-quarkus-langchain4j-ingest extension (shipped in Camel Quarkus 3.39.0; apache/camel-quarkus PRs #9018, #9040, #9078), ported to plain Camel with runtime-neutral naming. Once released, the Quarkus extension will delegate to this component, so both runtimes share a single implementation and Camel Quarkus keeps only its build-time developer experience — one engine, maintained once. A working PoC exists with a full unit suite plus an integration test that ingests real files through the declarative API into a real Qdrant container with a real all-MiniLM-L6-v2 model and verifies that semantic questions sharing no vocabulary with the documents retrieve the right ones by meaning.
> *Scope and follow-ups*
> Proposed as Preview. Deliberately append-only in this increment — an edited document re-ingests alongside its old segments; replace/delete arrive with a later synchronising engine, for which the document-identity metadata laid down here is the foundation. Follow-ups tracked separately: contributing the kamelets to apache/camel-kamelets, a langChain4jRecursiveTokenizer for the Tokenizer SPI, and the Camel Quarkus delegation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)