You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "Lyjk0415 (via GitHub)" <gi...@apache.org> on 2026/01/24 12:14:43 UTC

[I] 请教一个关于流处理方向的问题 [seatunnel]

Lyjk0415 opened a new issue, #10391:
URL: https://github.com/apache/seatunnel/issues/10391

   env {
     job.mode = "STREAMING"
     parallelism = 2
     
   }
   
   source {
     TDengine {
       url = "jdbc:TAOS-RS://XXXX:6041/"
       username = "XXXX"
       password = "XXXX"
       database = "XXXX"
       stable = "XXXX"
       lower_bound = "2020-11-15T14:11:13.196+08:00"
       upper_bound = "2026-11-15T14:11:13.196+08:00"
       scan_interval = 10s
       start_mode = "timestamp"
       timestamp_column = "ts"
       plugin_output = "tdengine_result"
     }
   }
   
   transform {
     Sql {
       source_table_name = "tdengine_result"
       result_table_name = "tdengine_result_cast"
   
       query = """
         SELECT
           IFNULL(serial_number, '') AS serial_number,
           ts,
           IFNULL(log_value, '') AS log_value,
           CAST(IFNULL(is_monitor, 0) AS INT) AS is_monitor,
           CAST(IFNULL(log_type, 0) AS INT) AS log_type,
           CAST(IFNULL(mode, 0) AS INT) AS mode,
           IFNULL(identify, '') AS identify,
           IFNULL(remark, '') AS remark,
           IFNULL(tenant_id, '') AS tenant_id,
           IFNULL(create_by, '') AS create_by
         FROM tdengine_result
       """
     }
   }
   
   sink {
     Doris {
       fenodes = "XXXX:8030"
       username = "XXXX"
       password = "XXXX"
       database = "XXXX"
       table = "XXXX"
       enable_delete = false
       write_mode = "insert"
       sink.label-prefix = "test-cdc"
       sink.enable-2pc = false
   
       doris.config {
         format = "json"
         read_json_by_line = true
         max-filtered-ratio = 0.1
       }
     }
   }为什么我这样配置之后还是不会触发我的流式处理,数据同步完之后并不会根据我设置的触发时间点去从td里面获取数据然后同步到doris


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 请教一个关于流处理方向的问题 [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10391:
URL: https://github.com/apache/seatunnel/issues/10391#issuecomment-3795844127

   <!-- code-pr-reviewer -->
   Based on the source code analysis, TDengine Source connector only supports **batch mode**, not streaming:
   
   - `TDengineSource.java:82-84` returns `Boundedness.BOUNDED`, meaning it reads data once and finishes
   - Official docs (`docs/en/connectors/source/TDengine.md`) confirm this: stream feature is unchecked
   - `scan_interval`, `start_mode`, and `timestamp_column` are **not valid** TDengine connector options (see `TDengineSourceOptions.java`)
   
   To achieve continuous incremental sync, you can:
   - Use an external scheduler (Airflow/DolphinScheduler) to run batch jobs periodically with updated `lower_bound`/`upper_bound`
   - Leverage TDengine's subscription feature with other streaming tools
   
   Regarding InfluxDB: it also **does not support streaming** (`InfluxDBSource.java:70-72` returns `BOUNDED`).
   
   For true streaming CDC, consider connectors like MySQL CDC or Postgres CDC instead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 请教一个关于流处理方向的问题 [seatunnel]

Posted by "Lyjk0415 (via GitHub)" <gi...@apache.org>.
Lyjk0415 commented on issue #10391:
URL: https://github.com/apache/seatunnel/issues/10391#issuecomment-3795776019

   了解了,那我还想请教一下如果我想使用TDengin的source有什么方式可以实现流式处理。还有influxdb 时序库可以实现流式处理吗
   
   
   
   
   
   发自我的iPhone
   
   
   ------------------ 原始邮件 ------------------
   发件人: yzeng1618 ***@***.***&gt;
   发送时间: 2026年1月24日 20:37
   收件人: apache/seatunnel ***@***.***&gt;
   抄送: Lyjk0415 ***@***.***&gt;, Author ***@***.***&gt;
   主题: Re: [apache/seatunnel] 请教一个关于流处理方向的问题 (Issue #10391)
   
   
   
   yzeng1618 left a comment (apache/seatunnel#10391)
    
   The TDengine source connector only supports batch mode and does not support streaming processing. Even if you set job.mode = "STREAMING", the TDengine connector will still run in batch mode.
    
   —
   Reply to this email directly, view it on GitHub, or unsubscribe.
   You are receiving this because you authored the thread.Message ID: ***@***.***&gt;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 请教一个关于流处理方向的问题 [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10391:
URL: https://github.com/apache/seatunnel/issues/10391#issuecomment-3794570255

   The TDengine source connector only supports batch mode and does not support streaming processing. Even if you set job.mode = "STREAMING", the TDengine connector will still run in batch mode.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] 请教一个关于流处理方向的问题 [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10391:
URL: https://github.com/apache/seatunnel/issues/10391#issuecomment-3794564778

   Excuse me, what engine is it based on? Zeta or Flink?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] I would like to ask a question about stream processing. [seatunnel]

Posted by "LeonYoah (via GitHub)" <gi...@apache.org>.
LeonYoah commented on issue #10391:
URL: https://github.com/apache/seatunnel/issues/10391#issuecomment-3797478018

   Stream processing requires the source end to start a listening program to continuously pull data, and the source end supports breakpoint resume to ensure that each pull is not repeated. As far as I know, the time series database does not have such a capability, and it can only write in real time but not read in real time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org