[GitHub] [incubator-seatunnel] zhaomin1423 commented on issue #1270: Whether to support restful api (http) as input source？

[GitBox <gi...@apache.org>]

zhaomin1423 commented on issue #1270:
URL: https://github.com/apache/incubator-seatunnel/issues/1270#issuecomment-1053916304


   > thanks for your sharing.
   > 
   > My data sync works are always limited by some http interfaces ratelimiter, especially for incremental sync. A classical http interface which supports incremental sync contains below request params or variant:
   > 
   > ```
   > {
   >     "startTime": "yyyy-MM-dd HH:mm:ss",
   >     "endTime": "yyyy-MM-dd HH:mm:ss",
   >     "pageIndex": 1,
   >     "pageSize": 50
   > }
   > ```
   > 
   > For ratelimiter and request params limitions, I have to controll concurrency and handle annoying network failure to get accurate and quick sync result. So, my core work is split sync task, then execute concurrently, update sync offset(latest synced time interval), I also belives flink source framework can do that process better.
   > 
   > Http authorization requires every request provides credentials, and I have more than 2500 credential account. Unbalanced hot data can appear in any credential account and any time, I had started a sync job for every credential account and every http interface, which just for incremental scenarios. How to handle numerous http interface credentials now really blocks me.
   > 
   > Now, I'm taking advantage of akka excellent concurrency and xxl-job to refactoring sync framework, you can contact me by wechat which is also kalencaya.
   
   I am willing to work with you. My wechat is 602128569.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org