You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "DanielLeens (via GitHub)" <gi...@apache.org> on 2026/08/14 01:42:25 UTC

[I] [Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing [seatunnel]

DanielLeens opened a new issue, #11801:
URL: https://github.com/apache/seatunnel/issues/11801

   ### Search before asking
   
   - [x] I searched existing issues and found related document/RAG work, but no issue that specifically discusses an `anydoc`-backed document-to-Markdown parser backend.
   
   Related issues:
   
   - #10889
   - #9861
   - #10608
   - #9716
   - #10916
   - #10920
   
   ### Description
   
   SeaTunnel already has an active Knowledge Sync / RAG direction, and the current `dev` branch has document-oriented file source support:
   
   - `FileFormat` includes `MARKDOWN` and `PDF` read strategies.
   - `PdfReadStrategy` extracts PDF document elements through PDFBox and can append RAG metadata fields.
   - `MarkdownReadStrategy` parses Markdown into document elements and can append RAG metadata fields.
   - `markdown_rag_metadata_enabled` and `pdf_rag_metadata_enabled` already expose the RAG metadata switch at file-source option level.
   
   This issue proposes adding an optional `anydoc`-backed parser backend for document-to-Markdown conversion, then reusing the existing Markdown/document element path where possible.
   
   `anydoc` is a Rust library from Firecrawl that converts Word, PowerPoint, Excel, OpenDocument, RTF, EPUB, CSV, and text-based PDF files to GitHub-Flavored Markdown. It currently publishes Node.js, Python, Rust, and WebAssembly bindings, and its repository is MIT licensed:
   
   https://github.com/firecrawl/anydoc
   
   Why this may fit SeaTunnel:
   
   1. SeaTunnel's current PDF support is PDF-specific and PDFBox-based. It is useful, but it does not provide one shared document-to-Markdown path for Office documents, RTF, EPUB, CSV, and PDF.
   2. SeaTunnel already has a Markdown reader that can turn Markdown into structured rows, so a document-to-Markdown parser backend can avoid duplicating structure extraction logic per document format.
   3. Knowledge Sync needs stable document/chunk metadata and a predictable text representation before chunking, embedding, and vector sink lifecycle handling. Markdown is a reasonable intermediate contract for that layer.
   
   ### Suggested Scope
   
   Add an optional document parser backend, for example:
   
   ```hocon
   source {
     LocalFile {
       path = "/path/to/documents"
       file_format_type = "document"
       document_parser = "anydoc"
       document_output_format = "markdown"
       document_rag_metadata_enabled = true
     }
   }
   ```
   
   Possible implementation shape:
   
   1. Add a new `DOCUMENT` file format or a dedicated document parser option instead of overloading the existing `PDF` behavior.
   2. Let the parser convert supported documents to Markdown first.
   3. Reuse or share the Markdown document-element parsing path to emit rows with fields such as `element_id`, `element_type`, `heading_level`, `text`, `source_uri`, `document_id`, `chunk_id`, `chunk_index`, and `content_hash`.
   4. Keep existing `file_format_type = pdf` behavior unchanged for compatibility.
   5. Gate anydoc behind an explicit option so deployments do not take a native/external runtime dependency unless they opt in.
   
   ### Integration Concerns
   
   This should be treated as an integration proposal, not as a simple Maven dependency bump:
   
   - SeaTunnel is JVM/Maven based, while anydoc currently exposes Rust/Node/Python/WASM bindings rather than a normal Java artifact.
   - A PoC could use an external command adapter, but production support should define a clear runtime packaging story, failure semantics, concurrency model, and cross-platform support.
   - Scanned or image-only PDFs still require OCR. anydoc handles text-based PDFs locally but does not replace OCR.
   - Existing PDFBox-based `PdfReadStrategy` should remain available unless the community explicitly decides otherwise.
   
   ### Acceptance Criteria
   
   - A design is documented for where the parser backend belongs: file source format, transform, or shared document parser SPI.
   - Existing PDF and Markdown behavior remains backward compatible.
   - The supported format list is explicit, including the no-OCR limitation for scanned PDFs.
   - The emitted schema is aligned with current Markdown/PDF RAG metadata and the Knowledge Sync metadata contract.
   - Runtime dependency packaging is documented and tested for at least Linux and macOS, with clear behavior when the backend is unavailable.
   - Unit tests cover text-based PDF, DOCX or PPTX, Markdown handoff, unsupported/scanned PDF behavior, and error handling.
   - At least one e2e example shows document files being read, converted to Markdown/structured rows, chunked, and written to a downstream sink.
   
   ### Usage Scenario
   
   Enterprise RAG and knowledge indexing pipelines often receive mixed document sets in object storage or local/HDFS paths:
   
   - PDFs
   - DOC/DOCX
   - PPT/PPTX
   - XLS/XLSX
   - ODT/ODS/ODP
   - RTF
   - EPUB
   
   A SeaTunnel pipeline should be able to discover those documents, normalize them into Markdown or document elements, add stable document/chunk metadata, then continue through chunking, embedding, and vector sink lifecycle handling.
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11801:
URL: https://github.com/apache/seatunnel/issues/11801#issuecomment-5315843084

   Thanks for the PoC work in #11813. I re-checked the current `dev` file-source boundary before replying here.
   
   The current repo already has two stable contracts we should preserve:
   - `file_format_type = markdown` -> `MarkdownReadStrategy`
   - `file_format_type = pdf` -> `PdfReadStrategy` with optional RAG metadata
   
   So I agree with keeping anydoc as an explicit, opt-in integration rather than folding it into the existing PDF path or silently changing current behavior.
   
   For the first production slice, I would keep the runtime model conservative:
   1. use a managed executable boundary first;
   2. make it explicitly opt-in via config;
   3. fail fast with a clear availability / version / platform check at startup;
   4. define timeout, temp-file / cleanup, and concurrency limits explicitly;
   5. document that scanned / image-only PDFs still require OCR and are out of scope for this backend.
   
   I would not start with WASM as the primary production path yet. The current question is not only whether it can run, but how we package, observe, and debug it on SeaTunnel workers across the supported environments. A managed executable is easier to reason about as a first slice, while still leaving room for a later WASM or sidecar evolution if the community wants it.
   
   On the design-record location: keeping the design note in the published developer docs is reasonable for now as long as it stays clearly marked as an open proposal and is updated or removed when the runtime model in #11801 is accepted, replaced, or rejected. The current PoC PR is already moving in that direction.
   
   So my recommendation is:
   - keep #11813 as the compatibility / design PoC;
   - keep existing PDF and Markdown behavior unchanged;
   - narrow v1 to an opt-in managed-executable backend that converts supported documents to Markdown and then reuses the current Markdown row / RAG path;
   - once that boundary is settled, split the production implementation into smaller reviewable steps.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11801:
URL: https://github.com/apache/seatunnel/issues/11801#issuecomment-5299640659

   @davidzollo @DanielLeens I would like to explore this feature and seems interesting ot me. 
   I will start with a small PoC and design note instead of adding the dependency directly. I want to compare the external process and WASM integration paths and verify how the generated Markdown can reuse the current MarkdownReadStrategy and RAG metadata fields  
   The first proposal will keep the existing PDF and Markdown paths unchanged and define the parser lifecycle packaging failure handling and supported platform boundary before implementation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11801:
URL: https://github.com/apache/seatunnel/issues/11801#issuecomment-5299887192

   @davidzollo @DanielLeens I have a design PoC ready on #11813 
   
   It verifies that anydoc generated Markdown can reuse the current MarkdownReadStrategy while keeping document and chunk IDs based on the original source URI. Existing PDF and Markdown behavior is unchanged and no runtime dependency or config option is added
   
   The remaining decision is how the backend should run on workers such as external executable WASM or sidecar. I opened this only as a draft for that discussion
   
   Let me know and i can have full fledged changes


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][RAG] Add optional anydoc backend for document-to-Markdown parsing [seatunnel]

Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11801:
URL: https://github.com/apache/seatunnel/issues/11801#issuecomment-5305633552

   @davidzollo @DanielLeens The PoC in #11813 is now complete and stays dependency free. Before starting the production implementation I need agreement on two points: whether the worker should invoke a managed executable, WASM module, or sidecar, and whether the design record should remain in the published docs or only in this issue. My preference for the first slice is a managed executable with explicit availability checks, timeout, cleanup, and supported-platform boundaries. Let me know your thoughts! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org