[GitHub] [airflow] rodrigoWizeLine edited a comment on issue #4766: [AIRFLOW-XXX] add prefix to file match in GCS_TO_S3 operator

[GitBox <gi...@apache.org>]

rodrigoWizeLine edited a comment on issue #4766: [AIRFLOW-XXX] add prefix to file match in GCS_TO_S3 operator
URL: https://github.com/apache/airflow/pull/4766#issuecomment-470125966
 
 
   The #4546 only takes out the subfolder of the result returned array
   
   My change, look into the right subfolder for the specific file and not the whole bucket
    Take this as example
   
   My_bucket
   --Subfolder1
   -----myfile1
   -----myfile2
   --Subfolder2
   -----myfile3
   -----myfile4
   Right now the function will return
   
   My_Bucket/Subfolder1/
   My_Bucket/Subfolder1/myfile1
   My_Bucket/Subfolder1/myfile2
   My_Bucket/Subfolder2/
   My_Bucket/Subfolder2/myfile3
   My_Bucket/Subfolder2/myfile4
   
   So if you are looking into a inner subfolder will look into the whole bucket and if is too big it will take forever just to compare
   
   Also right now it will never match because you are trying to sync the full path in GCS and just the file name.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services