[GitHub] [airflow] potiuk commented on issue #25373: SimpleHttpOperator: have an option to load data by batches

[GitBox <gi...@apache.org>]

potiuk commented on issue #25373:
URL: https://github.com/apache/airflow/issues/25373#issuecomment-1198323423

   I am not sure what interface you are proposing and why. I am not sure if it can be useful without seeing the interface, but if you want to make a PR proposal - feel free to attempt it - but I think it might be difficult to design it in a useful way. Simple Http Operator has generally rather little use when it comes to parallel or massive requests., especially to download some largel amounts of data because you then have to pass the data to other tasks.
   
   Note that even today you can use @task decorator and either iterate over or parallelise requests and download data in batches using HttpHooks instead of trying to use it with a Simple HTTP Operator. IMHO it is far more efficient followin the pattern I described in this blog post - https://medium.com/apache-airflow/generic-airflow-transfers-made-easy-5fe8e5e7d2c2 . I assign you to it, but please take a look and consider if what you want to do is at all needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org