You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "bitifi2019 (via GitHub)" <gi...@apache.org> on 2026/08/14 02:56:01 UTC

[I] 多表同步时如何实现覆盖写入(清空目标表)?custom_sql 无法动态适配每个表名 [seatunnel]

bitifi2019 opened a new issue, #11802:
URL: https://github.com/apache/seatunnel/issues/11802

   环境信息
   SeaTunnel 版本:2.3.18
   Doris版本: 3.x
   Doris Connector 版本:connector-doris-2.3.18.jar
   部署方式:集群模式(Master/Worker)
   JDK 版本:1.8
   
   问题描述
   在使用 Doris Sink 进行多表同步时,需要实现“覆盖写入”(即每次任务启动前先清空目标表,再写入数据),以保证同步的幂等性。
   在单表同步时,我可以通过以下配置实现:
   
   hocon
   Doris {
       ...
       data_save_mode = "CUSTOM_PROCESSING"
       custom_sql = "TRUNCATE TABLE test.table1"
       sink.enable-delete = "true"
   }
   该配置能在每个 Stream Load 批次前执行指定的 TRUNCATE 语句,清空目标表,效果正常。
   
   但在多表同步场景中(例如使用 table_list 或通配符匹配多张表),custom_sql 是静态字符串,无法根据当前写入的表名动态生成对应的 TRUNCATE 语句。即使配置了 data_save_mode = "CUSTOM_PROCESSING",custom_sql 也只会执行固定的一条 SQL,导致其他表的数据无法被清理,旧数据残留。
   
   已尝试方案
   1.在 data_save_mode = "DROP_DATA"时 ,日志显示 DataSaveMode 仍为 APPEND_DATA,可能该参数未正确生效。请问data_save_mode = "DROP_DATA",是否需要什么参数配置才能生效。
   
   2.在schema_save_mode = RECREATE_SCHEMA 时,并没有drop ... create , 而是执行create if not exists table1, 此能否实现先drop table 然后再create if not exists, 或者说是否需要什么参数配置才能生效。


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] How to implement over‑write (clear the target table) during multi‑table synchronization? custom_sql cannot dynamically adapt to each table name. [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #11802:
URL: https://github.com/apache/seatunnel/issues/11802#issuecomment-5307587413

   Thanks for describing the Doris multi-table overwrite case. I checked the current `dev` save-mode path before replying.
   
   The current Doris sink does not treat this as one global pre-sync hook. Save-mode handling is table-scoped: `DROP_DATA` maps to truncating the current target table, `RECREATE_SCHEMA` maps to dropping and recreating the current target table, and `CUSTOM_PROCESSING` executes `custom_sql` for the current target table through the standard save-mode handler.
   
   For the multi-table case, the first thing I would try is table placeholders in the sink config rather than one fixed SQL literal, for example:
   
   ```hocon
   data_save_mode = "CUSTOM_PROCESSING"
   custom_sql = "TRUNCATE TABLE ${table_full_name}"
   ```
   
   or `${database_name}` / `${table_name}` if you need to build the target name manually. The current sink factory path does run per-table placeholder replacement on string options before each sink instance is created, and Doris does not exclude `custom_sql` from that replacement path.
   
   If you only need truncate-before-write for each current target table, `data_save_mode = "DROP_DATA"` is the simpler built-in path on current `dev`.
   
   If you still see it fall back to `APPEND_DATA`, please update the issue in English and add:
   1. the exact multi-table sink config you used;
   2. the startup logs that show `APPEND_DATA`;
   3. whether the target table name is static, `${table_name}`, or `${table_full_name}`.
   
   With that, maintainers can verify whether this is a config/path mismatch or a real Doris multi-table bug.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org