You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "wuchunfu (via GitHub)" <gi...@apache.org> on 2026/01/29 03:18:50 UTC

[I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

wuchunfu opened a new issue, #10414:
URL: https://github.com/apache/seatunnel/issues/10414

   ### Search before asking
   
   - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc., and support not to repeatedly transfer files that have already been transferred. Support immediate deletion after transfer, file expiration deletion and time, file backup and deletion after transfer, file scanning interval time, as well as support priority queue, queue size, queue buffer size, thread count and other operations
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3815798354

   Hello, I’d like to confirm the specific requirements with you. Please feel free to supplement any details if there are any discrepancies on our understanding.
   
   -  you want a real-time/continuous file ingestion capability for file sources (FTP/SFTP/HDFS/local, etc.) that (a) periodically scans/monitors a directory for new/updated files, (b) avoids re-processing files already transferred, and (c) supports post-actions like delete after transfer, backup then delete, delete by retention/expiration, plus tunables like scan interval, priority queue/queue size/buffer size, and thread/concurrency. Is that accurate? Are you targeting a long-running streaming job, or a “run periodically” batch job?
   
   - Current SeaTunnel status already have a limited incremental sync mode sync_mode=update, but it is only supported for file_format_type=binary, and currently it is effectively only exposed/usable in the HdfsFile source (FTP/SFTP/local do not expose sync_mode=update options yet). Also, this is batch-style (file list is built at startup), not real-time monitoring.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3839622210

   1. Goal: add continuous discovery (keep scanning a directory/prefix) and reuse existing sync_mode=update to achieve “no repeated transfers / incremental sync”.
   2. Key idea: move discovery + incremental decision + split dispatch into the SourceSplitEnumerator; keep ReadStrategy as “read-only”.
   3. New common options (all file sources):
   - discovery_mode: once (default) / continuous
   - scan_interval: e.g. 10s (only for continuous)
   - (optional) start_mode: earliest / latest
   - Keep existing incremental configs: sync_mode=update + target_path + update_strategy + compare_mode (binary-only stays unchanged)
   4. Runtime flow:
   - Enumerator periodically scans path → builds FileCandidate(relativePath,len,mtime) → if sync_mode=update, compares against target_path (len/mtime/checksum) → enqueues only new/changed files into pending.
   - Reader calls sendSplitRequest() when idle; enumerator assigns from pending in handleSplitRequest().
   - Maintain inFlight to avoid re-dispatch on short intervals; Reader sends a SplitFinishedEvent to clear inFlight (file-level at-least-once).
   5. Semantics / scope:
   - No signalNoMoreSplits in continuous mode (long-running job; recommend streaming semantics).
   - No unbounded “seen” state (avoid ever-growing checkpoints); incremental skipping relies mainly on target-side comparison via sync_mode=update.
   ```mermaid
   sequenceDiagram
     autonumber
     participant U as User
     participant Eng as Engine
     participant E as Enumerator
     participant R as Reader
     participant FS as Source FS
     participant T as Target FS
   
     U->>Eng: submit job config
     Eng->>E: createEnumerator + open()
     Eng->>R: createReader + open()
   
     par continuous scanning
       loop every scan_interval
         E->>FS: listStatus(root)
         E->>E: build candidates(relativePath,len,mtime)
         alt sync_mode=update
           E->>T: getFileStatus/checksum(target_path/relative)
           E->>E: decider.shouldProcess()
         end
         E->>E: enqueue pending splits
       end
     and reading
       loop
         R->>E: sendSplitRequest() when idle
         E-->>R: assignSplit(pending...)
         R->>FS: read split via ReadStrategy
         R->>E: SplitFinishedEvent(fileKey)
         E->>E: remove inFlight(fileKey)
       end
     end
   
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "wuchunfu (via GitHub)" <gi...@apache.org>.
wuchunfu commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3817242203

   > [@wuchunfu](https://github.com/wuchunfu) Hello, I’d like to confirm the specific requirements with you. Please feel free to supplement any details if there are any discrepancies on our understanding.
   > 
   > * you want a real-time/continuous file ingestion capability for file sources (FTP/SFTP/HDFS/local, etc.) that (a) periodically scans/monitors a directory for new/updated files, (b) avoids re-processing files already transferred, and (c) supports post-actions like delete after transfer, backup then delete, delete by retention/expiration, plus tunables like scan interval, priority queue/queue size/buffer size, and thread/concurrency. Is that accurate? Are you targeting a long-running streaming job, or a “run periodically” batch job?
   > * Current SeaTunnel status already have a limited incremental sync mode sync_mode=update, but it is only supported for file_format_type=binary, and currently it is effectively only exposed/usable in the HdfsFile source (FTP/SFTP/local do not expose sync_mode=update options yet). Also, this is batch-style (file list is built at startup), not real-time monitoring.
   
   @yzeng1618 Thank you for your reply. You understand without a problem. We need a regular or monitored directory, and files that have already been read or transferred do not need to be transferred again. If you are interested, I can assign them to you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "DanielCarter-stack (via GitHub)" <gi...@apache.org>.
DanielCarter-stack commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3815857075

   <!-- code-pr-reviewer -->
   Thanks for the feature request. I've reviewed the codebase and need some clarification to better understand the scope:
   
   **Current state:**
   - All file connectors (FTP, SFTP, HDFS, LocalFile) operate in **batch mode** (`Boundedness.BOUNDED`) - see `BaseMultipleTableFileSource.java:62-63`
   - File lists are built once at startup in `FileSourceSplitEnumerator.java:79-83`, with no continuous monitoring
   - `sync_mode=update` exists for incremental sync (`FileBaseSourceOptions.java:133-142`) but:
     - Only supports `file_format_type=binary`
     - Only exposed for HDFS (`HdfsFile.md:83-87`)
     - Still batch-oriented, not real-time monitoring
   
   **Questions:**
   1. Do you require a **long-running streaming job** with continuous monitoring (requires `UNBOUNDED` mode architecture changes), or is **external scheduling** (e.g., crontab/Airflow running periodic batch jobs) acceptable?
   
   2. Would extending `sync_mode=update` to FTP/SFTP/local files and supporting all file formats meet your "avoid duplicate transfers" requirement?
   
   3. Which features are **must-have** vs. **nice-to-have**: periodic scanning, avoiding duplicates, post-transfer delete/archive, priority queues, concurrency controls?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "yzeng1618 (via GitHub)" <gi...@apache.org>.
yzeng1618 commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3822082321

   @wuchunfu Thanks for confirming and for the offer to assign this to me. My understanding matches: we need periodic/continuous directory scanning, and already-read/transferred files should not be processed again.
   I suggest a 3-step plan:
   - dedup/incremental: reuse existing sync_mode=update (currently binary only) and expose it for FTP/SFTP/Local sources to address “no repeated transfers” first.
   - Continuous monitoring: add a continuous mode (e.g., scan_interval) to keep discovering and processing new files at runtime.
   - Operational features: add delete/backup/retention and define recovery/commit semantics to avoid data loss.
   Can the above three steps be implemented one by one?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "wuchunfu (via GitHub)" <gi...@apache.org>.
wuchunfu commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-3824104266

   > [@wuchunfu](https://github.com/wuchunfu) Thanks for confirming and for the offer to assign this to me. My understanding matches: we need periodic/continuous directory scanning, and already-read/transferred files should not be processed again. I suggest a 3-step plan:
   > 
   > * dedup/incremental: reuse existing sync_mode=update (currently binary only) and expose it for FTP/SFTP/Local sources to address “no repeated transfers” first.
   > * Continuous monitoring: add a continuous mode (e.g., scan_interval) to keep discovering and processing new files at runtime.
   > * Operational features: add delete/backup/retention and define recovery/commit semantics to avoid data loss.
   > 
   > If there are relevant suggestions, please supplement them with specific details.
   
   @yzeng1618 I think your approach is fine, and I believe you can handle this matter well. I look forward to your contribution to this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Feature][Connector-V2][File] Add real-time monitoring and reading of file types such as FTP, SFTP, HDFS, local files, etc [seatunnel]

Posted by "DanielLeens (via GitHub)" <gi...@apache.org>.
DanielLeens commented on issue #10414:
URL: https://github.com/apache/seatunnel/issues/10414#issuecomment-4222247592

   Thanks for the original feature request.
   
   I checked the current `dev` history and this is no longer just a proposal: continuous discovery for FTP/SFTP/Local/HDFS sources has already been partially implemented on `dev`, and the remaining post-action/retention work is being tracked in follow-up PRs.
   
   So the issue is now in the “feature landed in slices” stage rather than “waiting for first implementation”.
   
   If your main need was continuous discovery + avoiding repeated transfers, please check the latest `dev` behavior first. If the remaining gap is specifically delete/backup/retention semantics after successful sync, that looks like the smaller remaining follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org