You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "arjun-guptha (via GitHub)" <gi...@apache.org> on 2023/07/04 10:20:19 UTC

[GitHub] [seatunnel] arjun-guptha opened a new issue, #5019: Fetching latest records in Batch mode - Apache Seatunnel

arjun-guptha opened a new issue, #5019:
URL: https://github.com/apache/seatunnel/issues/5019

   ### Search before asking
   
   - [X] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement.
   
   
   ### Description
   
   I have successfully deployed Seatunnel in my system. I'm currently working on synchronizing data between the source and sink in MySQL. I'm running Seatunnel in batch mode, but it's currently retrieving all records from the source and dumping them into the sink table. I need to run this job every 30 minutes, but it should only fetch the latest records from the source, not all the records.
   
   Is there a way to configure Seatunnel to only retrieve the latest records from the source and dump them into the sink table in batch mode?
   
   ### Usage Scenario
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] arjun-guptha commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "arjun-guptha (via GitHub)" <gi...@apache.org>.
arjun-guptha commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1621114947

   Hi @liugddx Thanks for your response.
   I'm curious to know if the current released versions of this seatunnel supports incremental offline batch synchronization ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Fetching latest records in Batch mode - Apache Seatunnel [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1756497103

   This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] baobao2016 commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "baobao2016 (via GitHub)" <gi...@apache.org>.
baobao2016 commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1635537582

   DolphinScheduler好像也不支持动态传递sql的where条件的参数


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1620250894

   Pass the incremental value into sql every time it is scheduled. This can be solved by custom parameters


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1635561813

   > DolphinScheduler 好像也不支持动态传递sql的where条件的参数
   
   SeaTunnel has custom parameters.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] liugddx commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "liugddx (via GitHub)" <gi...@apache.org>.
liugddx commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1621247765

   > Hi @liugddx Thanks for your response. I'm curious to know if the current released versions of this seatunnel supports incremental offline batch synchronization ?
   
   This scenario requires the use of scheduling capabilities, DolphinScheduler might be a good choice
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [seatunnel] baobao2016 commented on issue #5019: Fetching latest records in Batch mode - Apache Seatunnel

Posted by "baobao2016 (via GitHub)" <gi...@apache.org>.
baobao2016 commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1635512121

   这个是很常用的功能,希望能添加这方面的支持


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Fetching latest records in Batch mode - Apache Seatunnel [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5019:
URL: https://github.com/apache/seatunnel/issues/5019#issuecomment-1767393995

   This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Fetching latest records in Batch mode - Apache Seatunnel [seatunnel]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5019: Fetching latest records in Batch mode - Apache Seatunnel
URL: https://github.com/apache/seatunnel/issues/5019


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org