You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Hisoka-X (via GitHub)" <gi...@apache.org> on 2025/08/18 08:21:40 UTC
[I] Write a document to introduce the entire RAG Ready data processing process. [seatunnel]
Hisoka-X opened a new issue, #9718:
URL: https://github.com/apache/seatunnel/issues/9718
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write a document to introduce the entire RAG Ready data processing process. [seatunnel]
Posted by "yuebaiv (via GitHub)" <gi...@apache.org>.
yuebaiv commented on issue #9718:
URL: https://github.com/apache/seatunnel/issues/9718#issuecomment-3809493077
Hi @Hisoka-X and maintainers,
I'm interested in contributing to this RAG documentation task. I have experience with data integration and RAG applications, and I'd like to write a comprehensive guide for using SeaTunnel in RAG data pipelines.
## Proposed Documentation Outline
I plan to create a detailed Chinese document (`docs/zh/introduction/rag-data-processing.md`) covering:
### 1. **RAG Fundamentals**
- What is RAG and why it matters
- Complete data processing pipeline overview
- SeaTunnel's role in the RAG workflow
### 2. **5 Real-World Scenarios with Full Configurations**
- **Scenario 1**: Enterprise document knowledge base (PDF/DOCX/MD → Milvus)
- **Scenario 2**: Database vectorization (MySQL → Qdrant)
- **Scenario 3**: Real-time document sync (CDC → Vector DB)
- **Scenario 4**: Multi-source data fusion (Web + Docs + DB)
- **Scenario 5**: Large-scale optimization with Lance format
### 3. **Advanced Topics**
- Hybrid indexing (dense + sparse vectors)
- Metadata filtering acceleration
- Document chunking best practices
- Embedding model selection guide
### 4. **Performance Optimization**
- Batch processing strategies
- Parallelism tuning
- Connection pooling
- Monitoring with Prometheus metrics
### 5. **Complete Example Project**
- Docker Compose setup (SeaTunnel + Milvus + Qdrant + Embedding service)
- Ready-to-run configurations
- Testing and validation scripts
### 6. **FAQ and Troubleshooting**
- Handling large files
- API rate limiting
- Vector updates (upsert mode)
- Query optimization
## Why This Documentation is Valuable
1. **Practical**: 5 complete configurations that users can run immediately
2. **Comprehensive**: Covers the entire pipeline from data ingestion to vector storage
3. **Modern**: Focuses on the hot topic of RAG and LLM applications
4. **Bilingual Ready**: Starting with Chinese, can be translated to English later
## Timeline
I can complete this documentation within **2 weeks** (by February 11, 2026) and submit a PR.
Could you please assign this issue to me? I'm excited to contribute to the SeaTunnel community!
**GitHub**: @yuebaiv
Looking forward to your feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write a document to introduce the entire RAG Ready data processing process. [seatunnel]
Posted by "inchang-ing (via GitHub)" <gi...@apache.org>.
inchang-ing commented on issue #9718:
URL: https://github.com/apache/seatunnel/issues/9718#issuecomment-5848079280
Hi maintainers, I'd like to work on this issue.
Note: @yuebaiv expressed interest back in January with a detailed outline, but 8 months have passed with no PR and the issue remains unassigned — so I'd like to pick it up. @yuebaiv if you're actively working on this, just say so and I'll gladly withdraw.
My plan: a bilingual (en + zh) doc introducing SeaTunnel's RAG-ready data processing pipeline, referencing the existing connectors (file sources with document parsing, transforms, vector-database sinks like Milvus/Qdrant) with runnable example HOCON configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write a document to introduce the entire RAG Ready data processing process. [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo closed issue #9718: Write a document to introduce the entire RAG Ready data processing process.
URL: https://github.com/apache/seatunnel/issues/9718
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Write a document to introduce the entire RAG Ready data processing process. [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #9718:
URL: https://github.com/apache/seatunnel/issues/9718#issuecomment-5852030118
Could you clarify the intended scope and location of this document? For example: should it be a single end-to-end guide under docs/en/ (e.g. docs/en/concept/ or a new docs/en/rag/ section) covering unstructured source ingestion, text chunking/splitting, Embedding/LLM transforms, and vector-store sinks (Milvus, Qdrant, Pinecone, Elasticsearch, etc.), with a complete runnable job config example? Also, should it be added to the sidebars of both the English and Chinese docs?
For contributors picking this up: please base the doc on features that already exist in the codebase (relevant transforms such as Embedding/LLM and the vector-database connectors), link to their existing connector/transform pages instead of duplicating parameter tables, and include a full example config that has been verified to run. Please comment here before starting so we can avoid duplicate work.
It would help to list the concrete components that make up 'RAG Ready' in the issue body so the scope is unambiguous — e.g. which sources, transforms, and sinks are considered part of the pipeline, and which SeaTunnel version the doc should target.
<!-- streview-comment:1343 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org