You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "davidzollo (via GitHub)" <gi...@apache.org> on 2026/01/31 01:17:03 UTC
[I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
davidzollo opened a new issue, #10425:
URL: https://github.com/apache/seatunnel/issues/10425
Based on the [tracking issue #5677](https://github.com/apache/seatunnel/issues/5677) and source code analysis, the following connectors have not yet implemented true multi-table support (i.e., missing `table_list` configuration or the ability to handle multiple distinct tables/schemas in a single source instance).
We need community help to implement these!
### Connectors to be implemented:
- [ ] **AmazonDynamoDB** (Missing `table_list` option)
- [ ] **Cassandra** (Missing `table_list` option)
- [ ] **InfluxDB** (Missing `table_list` option)
- [ ] **IoTDB** (Missing `table_list` option)
- [ ] **MongoDB** (Missing `table_list` option. Note: `cdc-mongodb` supports it, but batch `mongodb` source does not appear to.)
- [ ] **Neo4j** (Missing `table_list` option)
- [ ] **OpenMLDB** (Missing `table_list` option)
- [ ] **Pulsar** (Currently only supports `topic_patterns` for single schema/stream. Needs explicit multi-table support if applicable.)
- [ ] **Socket** (Single stream only)
- [ ] **TDengine** (Has `sub_tables`, but likely restricted to single Super Table schema. Needs verification if full multi-table support is possible.)
- [ ] **Hudi** (Source implementation not found in `seatunnel-connectors-v2/connector-hudi`. Needs to be implemented.)
### Reference
- **Feature Issue**: #5677
- **Implementation References**:
- `JdbcSource` and `JdbcSourceFactory` (Supports `TABLE_LIST`)
- `KafkaSource` (Supports `TABLE_LIST`)
- `PaimonSource` (Supports `CatalogOptions.TABLE_LIST`)
### How to contribute
1. Pick a connector from the list.
2. Comment on this issue to claim it.
3. Implement `SupportMultipleTable` logic (usually involves adding `OptionRule` for `TABLE_LIST` and updating the Source to handle multiple `CatalogTable`s).
4. Submit a PR!
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4358721323
Thanks for refreshing the source tracker.
I rechecked the current multi-table source direction against the latest `dev` state, and the updated body is now much closer to the real progress:
- Pulsar / Redis / RocketMQ are already marked done
- RabbitMQ is still clearly linked to active work
- the remaining TODO rows are easier for new contributors to identify safely
That makes this issue much more useful as the coordination entry point for the remaining source connectors.
One small but important implementation reminder for future contributors: for new multi-table source work, please prefer `tables_configs` rather than the deprecated `table_list` path.
If more connectors land or new PRs are opened, please keep the tracker body synchronized in the same way. Thanks again for maintaining it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "Anthippi (via GitHub)" <gi...@apache.org>.
Anthippi commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3829297857
Hi @davidzollo, I would like to implement the RabbitMQ multi-table source feature. I have checked the requirements and I will start working on it. Please assign this to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "NixonWahome (via GitHub)" <gi...@apache.org>.
NixonWahome commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4467677625
Hi @davidzollo, I'd like to implement the Cassandra multi-table source feature if it is still open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4233385653
Thanks for refreshing the source tracker.
I rechecked the current multi-table source path, and the updated body is now much closer to the real `dev` progress:
- RocketMQ is already marked done
- RabbitMQ / Redis / Pulsar are clearly tied to active `dev` PRs
- the remaining TODO rows are easier to identify for new contributors
One small but important implementation reminder for future contributors: for new multi-table source work, please prefer `tables_configs` rather than the deprecated `table_list` path.
This issue is still a good coordination point for the remaining source connectors. Thanks again for keeping it synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5635791390
Confirmed from the remote PR records: Cassandra #10896, Pulsar #10825, Redis #10601, and RocketMQ #10619 are all merged into `dev`, so their tracker rows should remain Done.
Two corrections to the proposed reopen list: InfluxDB is already merged through #12247, and IoTDB is actively tracked by open PR #12246 with a successful Build but still requiring review. Neither is available for a parallel claim.
For Amazon DynamoDB, Elasticsearch, Hudi, MongoDB batch, Neo4j, and OpenMLDB, I found no matching open multi-table-source PR in the complete current PR search. That is not by itself a release for claiming: please audit the current connector source, recent merged work, and the tracker body for each exact connector before opening a new slice. Keep one connector per PR, use `tables_configs`, and link the resulting PR here. No labels or formal assignees were changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "JeremyXin (via GitHub)" <gi...@apache.org>.
JeremyXin commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3834118934
assign redis to me, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3827379538
> .take
@vbhavh which connector would you like to implement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "vbhavh (via GitHub)" <gi...@apache.org>.
vbhavh commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3828089607
dynamodb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5650910226
@Anthippi Yes — the RabbitMQ multi-table source (#10557) is merged into `dev`; the merge commit is 556dd79554a3c4a7f3216f7786e95bad3c9066fa. Current `dev` carries the RabbitMQ source `TABLE_CONFIGS` rule, produced catalog tables, queue-to-table initialization, and per-split row table-ID routing, so that connector is complete.
That means RabbitMQ is not an open claim on this umbrella. Its tracker row should be recorded as Done with #10557 as the evidence link, and no new RabbitMQ multi-table implementation should be started here.
If you were looking to pick something up, please check the tracker body for a row that is still marked TODO and has no linked PR, and post here before starting so we avoid duplicate work.
<!-- streview-comment:1016 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5743488721
@davidzollo Could we refresh the tracker below?
### Connectors to be implemented
| Type | Connector | Contributor | Status | PR |
| :--- | :--- | :--- | :--- | :--- |
| Source | Cassandra | @NixonWahome | Done | #10896 |
| Source | InfluxDB | @goutamadwant | Done | #12247 |
| Source | IoTDB | @goutamadwant | Done | #12246 |
| Source | Neo4j | @goutamadwant | Done | #11869 |
| Source | OpenMldb | @goutamadwant | Done | #12303 |
| Source | Pulsar | @onceMisery | Done | #10825 |
| Source | RabbitMQ | @Anthippi | Done | #10557 |
| Source | Redis | @JeremyXin | Done | #10601 |
| Source | RocketMQ | @kuleat | Done | #10619 |
| Source | Elasticsearch | @FuYouJ | Pending scope clarification: existing `index_list` supports multiple indexes | — |
| Source | Hudi | @LiJie20190102 | Pending scope clarification: current connector has sink/catalog support but no source implementation | — |
| Source | AmazonDynamoDB | @vbhavh | Open implementation gap; previous claim needs confirmation | No matching PR found |
| Source | MongoDB (Batch) | @chl-wxp | Open implementation gap; previous claim needs confirmation | No matching PR found |
@chl-wxp @vbhavh Are your MongoDB batch and DynamoDB implementations still active? I’d like to pick up any available slices using `tables_configs`, without duplicating ongoing work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-6051286564
Thanks @goutamadwant for putting the refreshed table together — a few suggestions for whoever updates the tracker.
Rows this thread already supports as Done: Cassandra (#10896), InfluxDB (#12247), Pulsar (#10825), Redis (#10601), RocketMQ (#10619) and RabbitMQ (#10557). @Anthippi, to close out your question: RabbitMQ is merged into `dev` as 556dd79554a3c4a7f3216f7786e95bad3c9066fa, so no new RabbitMQ multi-table source work should be started from this umbrella.
Rows I'd like confirmed before they are recorded as Done: IoTDB (#12246), Neo4j (#11869) and OpenMldb (#12303). IoTDB and OpenMldb were only announced here as prepared, and Neo4j hasn't been discussed in this thread beyond the table row. Could you reply with the current merge state of each?
AmazonDynamoDB should no longer read "No matching PR found" now that you've picked up that slice in #12665; I'd suggest recording it as "In progress" with you as contributor. Please keep the implementation in that single PR using `tables_configs`, and link back here when it is ready for review.
MongoDB (Batch) stays an open gap pending confirmation from the previous claimant; Elasticsearch and Hudi stay "Pending scope clarification" with the notes you wrote. If MongoDB (Batch) gets no response in this thread, please say so here before opening a slice for it so we avoid duplicate work.
<!-- streview-comment:1596 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3904720765
> [@davidzollo](https://github.com/davidzollo) [@DanielCarter-stack](https://github.com/DanielCarter-stack) I've been working on the Multi-Table implementation for the RabbitMQ Source. I have successfully implemented the discovery phase and split assignment, but I am currently blocked by a `NullPointerException` during the data collection phase. When deserializing messages using `JsonDeserializationSchema`, the produced `SeaTunnelRow` has a null `rowType`. In Multi-Table mode, the engine calls SeaTunnelRow.getBytesSize(), which leads to this crash:
>
> java.lang.NullPointerException: Cannot invoke "org.apache.seatunnel.api.table.type.SeaTunnelRowType.getFieldType(int)" because "rowType" is null
> at org.apache.seatunnel.api.table.type.SeaTunnelRow.getBytesSize(SeaTunnelRow.java:122)
> at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:103)
> I have updated the `SourceReader` to maintain a map of `DeserializationSchema` and `SeaTunnelRowType` per queue. I am explicitly setting `record.setTableId(queueName)` inside the collector. Despite setting the tableId, the engine still requires rowType for size calculation.
>
> The main issue is that while I have the correct `SeaTunnelRowType` available in the Reader, I haven't found a way to pass it into the `SeaTunnelRow` object. The class lacks a public `setRowType` method, and the `JsonDeserializationSchema` only populates the data fields, leaving the type information null.
>
> I would like to ask:
>
> * What is the intended way to bind `SeaTunnelRowType` to the row in a Multi-Table source before it is collected?
In Multi-Table mode, **SeaTunnelRowType is not intended to be bound directly to each Row object.**
### Intended Way
SeaTunnel's design separates type information from Row data, associating them via `TableId`.
1. **Collector Maintains the Mapping**:
The `SeaTunnelSourceCollector` internally maintains a `Map<String, SeaTunnelRowType> rowTypeMap`. This map stores the mapping from `tableId` to `SeaTunnelRowType`. This information comes from the list of `CatalogTable`s registered during Source initialization.
2. **Row Carries the ID**:
Each `SeaTunnelRow` object only needs to carry the correct `tableId` (via `row.setTableId(String id)`).
3. **Runtime Binding**:
When the Collector receives data and needs to calculate the size (`getBytesSize`) or perform other operations, it uses `row.getTableId()` to look up the corresponding `RowType` in the `rowTypeMap`, and then passes it to the generic `SeaTunnelRow` methods for calculation.
**Summary**: You do not (and cannot) inject `RowType` into `SeaTunnelRow`. The only thing you need to do is ensure that the `TableId` string set by the data producer (Reader/Deserializer) **matches exactly** the `TableId` string registered on the Engine side (Collector). If you encounter an NPE, it is because these two strings do not match (e.g., one has a database prefix and the other doesn't; or as we found earlier, one is a normal string and the other is the `toString` representation of an Optional).
> * Is this a known issue with the current `JsonDeserializationSchema` implementation for multi-table scenarios?
We didn't find this problem. You can help fix it, thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "Anthippi (via GitHub)" <gi...@apache.org>.
Anthippi commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5635970223
Is RabbitMQ merged into dev?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5658214121
@Anthippi — yes, RabbitMQ multi-table source is merged into `dev`. The merge record is PR #10557, landed on 2026-05-06 as commit 556dd79554a3c4a7f3216f7786e95bad3c9066fa. Current `dev` already carries the RabbitMQ source `TABLE_CONFIGS` rule, produced catalog tables, queue-to-table initialization, and per-split row table-ID routing, so there is nothing left to implement for that connector under this umbrella.
Remaining asks on this point:
- The tracker body should record the RabbitMQ row as Done with #10557 as the evidence link — if anyone sees it still listed as in-progress or open, please flag it here so it can be corrected.
- Please do not start a new RabbitMQ multi-table source slice from this issue; any follow-up on that connector should go through a separate bug/enhancement issue against the merged implementation instead.
<!-- streview-comment:1029 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5747418955
Thanks @goutamadwant for picking up the OpenMLDB slice and linking #12303 here — that keeps the umbrella easy to follow.
A couple of things on the OpenMLDB row before the tracker body is refreshed:
- In your proposed table, OpenMldb is listed as `Done` with #12303. From this thread alone I can't tell whether #12303 has actually merged into `dev` or is still under review. Could you confirm the current state of #12303? If it is still open, the row should read as in-progress with the PR linked, and we'll flip it to Done once it lands.
- Please confirm the implementation follows the same pattern as your IoTDB/InfluxDB work: `tables_configs` (not the deprecated `table_list` path), one connector in the PR, and legacy single-table config still working.
- If #12303 does not already reference this umbrella issue in its description, please add the link so the cross-reference shows up on both sides.
Once you confirm the merge state, I'm happy to have the OpenMLDB row updated accordingly.
<!-- streview-comment:1184 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "davidzollo (via GitHub)" <gi...@apache.org>.
davidzollo commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3827072249
If there is anybody who'd like to implement these connector features, please leave a related message like `I'd like to implement XX multi-table source` and then you can submit your PR within four weeks.
A very kindly reminder: Other guys can pick it up if the related PR is not submitted within four weeks. ^-^
Here are some helpful resources to get started:
- [Contribution Setup Guide](https://seatunnel.apache.org/docs/contribution/setup)
- [Code Submission Guide](https://seatunnel.apache.org/community/submit_guide/submit-code)
By the way, since this might be your first contribution, I think we can have a more in-depth communication to help you get familiar with SeaTunnel. Feel free to contact me on LinkedIn ([David Zollo](https://www.linkedin.com/in/davidzollo)) or WeChat ID(davidzollo). When adding me, please let me know your GitHub ID. I'd be happy to assist you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "Anthippi (via GitHub)" <gi...@apache.org>.
Anthippi commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3898791124
I've been working on the Multi-Table implementation for the RabbitMQ Source. I have successfully implemented the discovery phase and split assignment, but I am currently blocked by a `NullPointerException` during the data collection phase.
When deserializing messages using `JsonDeserializationSchema`, the produced `SeaTunnelRow` has a null `rowType`. In Multi-Table mode, the engine calls SeaTunnelRow.getBytesSize(), which leads to this crash:
```java
java.lang.NullPointerException: Cannot invoke "org.apache.seatunnel.api.table.type.SeaTunnelRowType.getFieldType(int)" because "rowType" is null
at org.apache.seatunnel.api.table.type.SeaTunnelRow.getBytesSize(SeaTunnelRow.java:122)
at org.apache.seatunnel.engine.server.task.SeaTunnelSourceCollector.collect(SeaTunnelSourceCollector.java:103)
```
I have updated the `SourceReader` to maintain a map of `DeserializationSchema` and `SeaTunnelRowType` per queue. I am explicitly setting `record.setTableId(queueName)` inside the collector. Despite setting the tableId, the engine still requires rowType for size calculation.
The main issue is that while I have the correct `SeaTunnelRowType` available in the Reader, I haven't found a way to pass it into the `SeaTunnelRow` object. The class lacks a public `setRowType` method, and the `JsonDeserializationSchema` only populates the data fields, leaving the type information null.
I would like to ask:
- What is the intended way to bind SeaTunnelRowType to the row in a Multi-Table source before it is collected?
- Is this a known issue with the current JsonDeserializationSchema implementation for multi-table scenarios?
I have the full implementation ready and would appreciate a tip on the correct way to bypass this NPE so I can test my code and submit the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "onceMisery (via GitHub)" <gi...@apache.org>.
onceMisery commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3976327610
> [@davidzollo](https://github.com/davidzollo) Hello! I'd like to implement Pulsar multi-table source!
@dybyte Can I take it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "dybyte (via GitHub)" <gi...@apache.org>.
dybyte commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3976862618
> > [@davidzollo](https://github.com/davidzollo) Hello! I'd like to implement Pulsar multi-table source!
>
> [@dybyte](https://github.com/dybyte) Can I take it ?
Please feel free to proceed with the implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5645799553
Yes. I verified the remote record: RabbitMQ multi-table source PR #10557 merged into dev on 2026-05-06 as 556dd79554a3c4a7f3216f7786e95bad3c9066fa. Current dev still has the RabbitMQ source TABLE_CONFIGS rule, produced catalog tables, queue-to-table initialization, and per-split row table-ID routing.
So RabbitMQ is not an open claim: its tracker row should be recorded as Done with #10557 as the evidence link. No new RabbitMQ multi-table implementation should be started from this umbrella. No labels or formal assignees were changed in this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "kuleat (via GitHub)" <gi...@apache.org>.
kuleat commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4067845024
assign RocketMQ to me, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3831160575
<!-- code-pr-reviewer -->
Thanks for raising this feature request. I've verified that the 14 listed source connectors (Cassandra, Elasticsearch, Hudi, InfluxDB, IoTDB, MongoDB, Neo4j, OpenMldb, Pulsar, RabbitMQ, Redis, RocketMQ, AmazonDynamoDB) are indeed missing multi-table support.
**Implementation references:**
- Kafka connector provides the standard pattern for `TABLE_CONFIGS` usage in `KafkaSourceFactory.optionRule()` and `KafkaSourceConfig.createMapConsumerMetadata()` (connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/source/)
- API definition: `TableSchemaOptions.TABLE_CONFIGS` in `seatunnel-api/src/main/java/org/apache/seatunnel/api/options/table/TableSchemaOptions.java` (L36-L42)
**Key requirements for implementation:**
1. Add `TABLE_CONFIGS` option to connector's `optionRule()` (preferably with `exclusive` constraint against single-table options)
2. Modify source constructor to accept `List<CatalogTable>`
3. Implement `getProducedCatalogTables()` to return multi-table metadata
4. Handle multiple table/collection/topic configurations in source reader initialization
**Note:** `TABLE_LIST` (deprecated, see `CatalogOptions.TABLE_LIST`) should not be used for new implementations.
For contributors: please reference the existing RabbitMQ (connector-rabbitmq/src/main/java/org/apache/seatunnel/connectors/seatunnel/rabbitmq/source/RabbitmqSourceFactory.java) and similar connectors as starting points.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4468816608
Thanks for volunteering.
From the current tracker state, the Cassandra multi-table source slice is still open. There is no active linked PR for that connector in this umbrella thread right now, so it is a reasonable item to pick up.
The safest way to move this forward is to keep the implementation focused on Cassandra only, and use this umbrella issue just as the coordination entry:
- one connector slice
- one reviewable PR
- keep the behavior aligned with the existing multi-table source pattern already used by other connectors
If you start on it, please link the PR back to this issue so the tracker stays easy to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-4227770237
Thanks for keeping this tracker active. I checked the current `dev` state, and this umbrella is already in active progress: RocketMQ multi-table source has merged in PR #10619, and RabbitMQ / Redis / Pulsar still have open `dev` PRs (#10557, #10601, #10605).
So the main gap now is less about feature direction and more about keeping the tracker body synchronized with what has already landed vs what is still in progress.
If someone wants to pick a remaining connector, this issue is still the right place to coordinate. It would just help to refresh the checklist first.
Thanks again to everyone pushing the connector-by-connector work here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5629232971
The status table shows Cassandra (#10896), Pulsar (#10825), Redis (#10601), and RocketMQ (#10619) as Done — could a maintainer confirm those PRs are merged and update the table accordingly? Also, please check whether the in-progress items (AmazonDynamoDB, Elasticsearch, Hudi, MongoDB Batch) still have active contributors or should be reopened for claiming.
InfluxDB, IoTDB, Neo4j, and OpenMldb are still unassigned. Anyone interested: please comment here with the connector name before starting so a maintainer can assign it and avoid duplicate work. Use `ConnectorCommonOptions.TABLE_CONFIGS` (`tables_configs`), not the deprecated `table_list`, and include e2e tests covering multiple tables/schemas in one source instance.
For contributors: please link your PR in this issue and reference this issue number in the PR description so the tracking table can be kept in sync. Good implementation references are `JdbcSource`/`JdbcSourceFactory`, `KafkaSource`, and `PaimonSource`.
<!-- streview-comment:963 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "vbhavh (via GitHub)" <gi...@apache.org>.
vbhavh commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-3827144183
.take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "SEZ9 (via GitHub)" <gi...@apache.org>.
SEZ9 commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5643077721
Thanks everyone for keeping this umbrella moving.
@goutamadwant — thanks for taking both IoTDB and InfluxDB with `tables_configs` plus the Java 8/11 and legacy-compatibility coverage. Based on the latest tracker check in this thread, InfluxDB is already merged and the IoTDB PR is open and still waiting on review, so please make sure the IoTDB PR is linked back to this issue and flag here once it is ready for a final pass. I'll update the checklist rows accordingly.
@Anthippi — I can't confirm a RabbitMQ merge from this thread; the last status recorded here was that RabbitMQ was still tied to active work rather than marked Done. I'll verify the current state and refresh the tracker body. If you are asking because you'd like to pick it up, please comment here first so we avoid duplicate work.
For anyone else looking for an open slice: with InfluxDB and IoTDB now covered, the remaining unassigned rows from this tracker are Neo4j and OpenMldb. Please claim one here before starting, keep it to one connector per PR, use `tables_configs` (not the deprecated `table_list`), follow the existing multi-table source pattern, and link the PR to this issue.
<!-- streview-comment:987 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5656838284
Prepared OpenMldb multi-table support #12303
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-5613654456
@DanielLeens I've prepared both the IoTDB -#12246 and InfluxDB - #12247 multi-table source implementations using `tables_configs`, with Java 8/11 validation and legacy compatibility coverage
Let me know if there are any comments or suggestions. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-6031731865
I would like to work on the AmazonDynamoDB multi-table source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Umbrella][Feature][Connector] Need help for supporting multi-table source feature [seatunnel]
Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10425:
URL: https://github.com/apache/seatunnel/issues/10425#issuecomment-6037720380
Thanks for picking up the AmazonDynamoDB source slice. I checked current `dev` before treating this as available work: the source factory still requires one `table` and `schema`, `AmazonDynamoDBSource` produces one `CatalogTable`, and the earlier #10497 work is multi-table **sink** support only. I also checked open PRs and found your #12665, so there is no separate competing source implementation to start.
The PR's direction is the right one for this umbrella: use `tables_configs` for independently declared tables/schemas, keep connection options at source scope, carry table identity through splits and emitted rows, and preserve existing single-table checkpoint restore semantics. The reader's duplicate-page fix should stay explicitly tested as a separate behavioral correction. Please keep the feature in #12665 rather than opening another implementation PR; its Build is successful but it still needs a full compatibility, split/state, routing and E2E review before the tracker can mark the source complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org