You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "countradooku (via GitHub)" <gi...@apache.org> on 2026/06/08 16:08:18 UTC
[I] feat(connectors): add SurrealDB source connector [iggy]
countradooku opened a new issue, #3439:
URL: https://github.com/apache/iggy/issues/3439
## Description
Apache Iggy should support SurrealDB as a first-class source connector.
SurrealDB is a multi-model database with document, graph, relational, time-series, geospatial, full-text, vector, hybrid search, and realtime capabilities. Its Rust SDK supports remote connections, async querying, authentication, and live queries. SurrealDB also exposes change feeds for replaying table changes, which makes it a good fit for ingesting database changes into Iggy streams.
This issue tracks the SurrealDB source only. The SurrealDB sink is tracked separately in #3438 so the write path and read/changefeed path can be designed, reviewed, and tested independently.
Affected area / component: Connectors
## Proposed solution
Add `core/connectors/sources/surrealdb_source`.
Expected source support:
- Remote SurrealDB endpoint configuration.
- Namespace and database selection.
- Credential fields stored as `SecretString` and never logged.
- `polling`, `live`, and `changefeed` modes where supported by SurrealDB.
- Configurable tables and custom SurrealQL query.
- Persistent connector state for cursor/versionstamp/timestamp tracking.
- Create/update/delete operation envelopes for realtime/changefeed modes.
- Configurable payload extraction and metadata wrapping.
- Payload formats: `json`, `text`, and `raw` where practical.
- Recovery behavior that avoids duplicate delivery where the backend exposes enough cursor information.
- Configurable batch size, poll interval, retry count, retry delay, and `verbose_logging`.
- Proper transient/permanent error mapping.
Implementation requirements:
- Follow existing connector layout and naming conventions.
- Keep the plugin crate as `cdylib` plus `lib`.
- Add README and example runtime config files.
- Add unit tests for config validation, payload extraction, operation envelope mapping, error classification, and state restore.
- Include the canonical source state tests expected by the connectors subsystem.
- Add Docker-backed integration tests for source polling and at least one live/query or changefeed path.
- Run `cargo fmt --all`, `cargo sort --no-format --workspace`, targeted tests, and connector-relevant CI scripts.
- Validate any new dependency through the repo's third-party license workflow before merging.
## Dependency note
Use the SurrealDB Rust SDK only after confirming it is acceptable for Apache Iggy's dependency policy. Avoid embedded SurrealDB storage features unless their licensing and transitive dependencies are explicitly reviewed.
## Alternatives considered
Using the existing HTTP sink or a generic HTTP polling source would not cover source state, live queries, change feeds, SurrealDB auth/session handling, operation envelopes, or recovery semantics.
Adding only a sink connector would be useful, but it would not cover SurrealDB's strongest streaming use case: moving database changes into Iggy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): add SurrealDB source connector [iggy]
Posted by "AmalenduManoj (via GitHub)" <gi...@apache.org>.
AmalenduManoj commented on issue #3439:
URL: https://github.com/apache/iggy/issues/3439#issuecomment-4670018835
can i work on this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): add SurrealDB source connector [iggy]
Posted by "countradooku (via GitHub)" <gi...@apache.org>.
countradooku commented on issue #3439:
URL: https://github.com/apache/iggy/issues/3439#issuecomment-4670063595
I am working on this right now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] feat(connectors): add SurrealDB source connector [iggy]
Posted by "justinmclean (via GitHub)" <gi...@apache.org>.
justinmclean commented on issue #3439:
URL: https://github.com/apache/iggy/issues/3439#issuecomment-5921808439
@countradooku checking in here as well as on #3496. Still working on this? If not I'll unassign it so someone else can pick it up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org