You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/08/11 03:30:54 UTC
[I] [Feature][Connector-V2] Object storage CDC (OSS/S3): event-notification driven discovery, changelog semantics, and on-demand content fetch for real-time AI/multimodal pipelines [seatunnel]
davidzollo opened a new issue, #11758:
URL: https://github.com/apache/seatunnel/issues/11758
### Search before asking
- [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
### Description
## Background
Alibaba Cloud's Realtime Compute for Apache Flink (VVR 11.8.0+) recently shipped an **OSS CDC connector** that turns object storage into a first-class CDC source for real-time AI / multimodal pipelines:
- Article: [Flink OSS CDC: 让AI多模态数据处理实时化](https://mp.weixin.qq.com/s/W8yZe4lie9lZ9qC8gyuUNA)
- Docs: https://help.aliyun.com/zh/flink/realtime-flink/developer-reference/oss-cdc-beta
Its core idea: instead of periodically LIST-scanning a bucket, OSS pushes **event notifications** into a lightweight message queue (MNS), and the connector consumes those events and converts object changes (create / overwrite / delete) into a **standardized changelog stream**. Key capabilities:
1. Full + incremental startup modes (`initial`, `snapshot`, `timestamp`, `latest-offset`, `earliest-offset`, `committed-offset`), with **ETag-based dedup** during the snapshot-to-incremental handover.
2. Checkpoint/savepoint recovery in the incremental phase.
3. Path filtering by object key prefixes, dynamic partition (directory) discovery via regex.
4. **Metadata passthrough**: key, URL, size, modification time, ETag.
5. **Lazy content fetching**: by default only metadata is emitted; file content is read only when the pipeline actually calls `FETCH_CONTENT(url)`.
6. Object-key regex to Table ID mapping (`oss-key.to.table-id.matching.pattern`) for multi-table transform/route.
7. In-pipeline AI functions (`AI_EMBED` / `AI_IMAGE_EMBED`) so a new image or an overwritten knowledge document is embedded and written to the vector store within seconds (image search, RAG freshness).
## What SeaTunnel already supports today (dev)
We are actually not far from this. Current `dev` already has a polling-based version of most of the ingestion story in `connector-file-base`, inherited by the object storage connectors (`OssFile` and `S3File` extend `BaseMultipleTableFileSource`, alongside `LocalFile` / `HdfsFile` / `FtpFile` / `SftpFile`; `CosFile` / `ObsFile` are not on this base yet):
- **Continuous file discovery** (#10563, #11312): `discovery_mode = continuous` makes the source UNBOUNDED (`BaseMultipleTableFileSource#getBoundedness`), with `ContinuousMultipleTableFileSourceSplitEnumerator` re-scanning the path every `discovery_interval` (default 10s), `start_mode = earliest | latest`, checkpoint-gated `post_sync_action` and retention cleanup.
- **Changed-file detection**: re-emits a file when its `(length, modificationTime)` version changes (`SplitVersion` in the continuous enumerator), reusing `sync_mode = update` comparison semantics (`len_mtime` / `checksum`).
- **Multimodal / AI building blocks**: BINARY format for raw file transfer, PDF parser + PDF/Markdown RAG metadata (#10105, #11571, #10844, #10964), Embedding transform with multimodal support (`seatunnel-transforms-v2` `nlpmodel/embedding/multimodal`, #9996), LLM transform, vector data types, and vector store sinks (Milvus, Qdrant, etc.).
So "keep a bucket continuously synced into an AI pipeline" is already possible today with polling. The remaining gaps are what this issue proposes to close.
## Proposed feature: object storage CDC capabilities
1. **Event-notification driven discovery.** Polling has a latency floor of `discovery_interval` and LIST cost grows with bucket size. Propose a pluggable event channel for object stores as an additional discovery mode (e.g. `discovery_mode = event`), starting with OSS event notification -> MNS and S3 event notification -> SQS (MinIO bucket notifications later), including an `initial`-style handover: enumerate existing objects first, then consume events, with dedup across the handover.
2. **Changelog semantics for object changes.** Today re-emitted changed files are plain appended rows: downstream cannot distinguish a new object from an overwritten one, and deletes are not captured at all. Propose optional changelog output: INSERT for new objects, UPDATE for overwrites, DELETE for removals, so upsert-capable sinks (Paimon, Iceberg, JDBC, vector stores) can keep derived data consistent — exactly the stale-chunk problem described in the RAG umbrella #10889.
3. **ETag/version-based change detection for object stores.** `(length, mtime)` misses same-size overwrites within mtime granularity; object stores return ETag for free in LIST responses. Add an `etag` compare mode where the filesystem supports it.
4. **Metadata-only emission with on-demand content fetch.** Emit object metadata (key, URL, size, mtime, ETag) without reading content, and fetch content lazily only when a downstream transform actually needs it. This keeps one pipeline usable both for lightweight reference-only sync and heavy vectorization jobs. Relates to #9737 (metadata columns for file source).
5. **`timestamp` start mode.** `start_mode` currently supports only `earliest` / `latest`; add starting from a given timestamp.
6. **Object-key regex to table mapping.** Derive table identity from key regex capture groups (analog of `oss-key.to.table-id.matching.pattern`) so one source can feed multi-table transform/route without enumerating `tables_configs` manually.
7. **Docs + E2E for object storage continuous mode.** `discovery_mode` is currently documented for `FtpFile` / `HdfsFile` / `LocalFile` / `SftpFile` only; object storage connector docs (`OssFile`, `S3File`, ...) do not cover it yet. Document and E2E-verify the object storage path as part of this work.
Items 1-4 are the core; 5-7 are incremental follow-ups. Happy to split this into subtasks under a design discussion if there is interest — implementation-wise everything can build on the existing `connector-file-base` continuous discovery rather than a brand-new connector.
### Usage Scenario
- E-commerce: product image uploaded to a bucket -> embedded via multimodal model -> written to a vector store within seconds; image search / similar-item recommendation / content moderation see new items immediately.
- Enterprise RAG knowledge base: a policy document overwritten in the bucket is detected (ETag change), re-parsed, re-embedded and upserted, and deleted documents are cleaned up, so the AI assistant never answers from a stale version.
- General multimodal lakehouse ingestion: model weights, audio/video, documents landed in object storage flow into Paimon/Iceberg/vector stores in near real time without full rescans.
### Related issues
- #10889 (umbrella: production-grade Knowledge Sync for RAG — this issue is the real-time ingestion head of that story)
- #9713 (connector prepare for RAG)
- #9737 (metadata columns for file source)
- #10608 (PDF parser / multimodal labeling operators)
- #10563, #11312 (existing polling-based continuous discovery this proposal builds on)
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector-V2] Object storage CDC (OSS/S3): event-notification driven discovery, changelog semantics, and on-demand content fetch for real-time AI/multimodal pipelines [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11758:
URL: https://github.com/apache/seatunnel/issues/11758#issuecomment-5262378068
Thanks for the details @davidzollo I would like to work on this in stages.
For the first PR, I plan to:
- Expose the existing polling-based continuous discovery options for `S3File` and `OssFile`
- Add support for `discovery_mode`, `scan_interval`, and `start_mode` in the source factories
- Keep the existing bounded-source behavior unchanged by default
- Add focused configuration and continuous-discovery tests
- Update the English and Chinese documentation
For event notifications, changelog semantics, checkpoint deduplication, ordering, and lazy content fetching, I think we should agree on the design through a separate STIP before implementation. What do you propose?
I will have a draft PR for the first slice ready soon. Please let me know if this split looks reasonable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector-V2] Object storage CDC (OSS/S3): event-notification driven discovery, changelog semantics, and on-demand content fetch for real-time AI/multimodal pipelines [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #11758:
URL: https://github.com/apache/seatunnel/issues/11758#issuecomment-5275865312
@DanielLeens I have the polling-based first slice ready for review
#11789
It exposes the existing continuous discovery and update options for S3File and OssFile while keeping `discovery_mode=once` as the default
I also added factory validation tests bilingual docs and a MinIO E2E test that verifies an object uploaded after the job starts is discovered and copied
This slice does not include event notifications delete event propagation changelog records or ETag deduplication those still need a separate design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector-V2] Object storage CDC (OSS/S3): event-notification driven discovery, changelog semantics, and on-demand content fetch for real-time AI/multimodal pipelines [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11758:
URL: https://github.com/apache/seatunnel/issues/11758#issuecomment-5266691191
Thanks for proposing a staged path.
I re-checked the current `dev` implementation before replying here. The first slice you described matches the gap we actually have today: the file-base layer already has the continuous-discovery primitives (`discovery_mode`, `scan_interval`, `start_mode`), but the `S3File` and `OssFile` source factories still do not expose that contract cleanly.
So I think your split is reasonable. For phase 1, please keep the boundary explicit:
1. expose the existing polling-based continuous discovery path for `S3File` and `OssFile` only;
2. preserve the current bounded-source behavior by default;
3. keep the docs and tests explicit that the current continuous path still requires `sync_mode=update` and `file_format_type=binary`;
4. do not mix in event notifications, changelog/delete semantics, ordering guarantees, checkpoint handover deduplication, or lazy content fetch.
For those later capabilities, I agree they should go through a separate STIP because they change the source contract rather than just surfacing an existing one. If you open the first PR with that narrower scope, maintainers can review it independently and then use a STIP to discuss the broader OSS/S3 CDC model.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org