[GitHub] [airflow] potiuk commented on issue #21218: Enable specify the hook on GenericTransfer

[GitBox <gi...@apache.org>]

potiuk commented on issue #21218:
URL: https://github.com/apache/airflow/issues/21218#issuecomment-1030838373


   This is not possible, because Hooks when instantiated might be created with different init parameters.
   
   Also I think the 'GenerlcTransfer` is not that useful and I think it adds unnecessary complexity and generalization where we already have everything needed to support the "generic transfer" by writing decorated tasks and using hooks.
   
   I think this is much nicer and more "new Airflow-y" way of implementing transfer tasks using precisely the combination of Hooks and connections you need. It's much more explicit:
   
   ```
   @task
   def transfer_task():
      s3_hook = S3Hook(....params here ...)
      gcs_hook = GCSHook(... params here ...)
      // use .upload()/download() or streaming capabilities of the hooks to perform the transfer
   
   ```
   
   I will convert it into discussion and would love to hear what others think, but I believe we should simply describ this as a valid pattern to do transfer rather than investing into `GenericTransfer`.
   
   WDYT @BasPH @dstandish @ashb @kaxil @mik-laj  @turbaszek @olchas  (I think at some point of time you were involved in similar discussions). 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org