[GitHub] [airflow] potiuk edited a comment on issue #16148: Downloading files from S3 broken in 2.1.0

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #16148:
URL: https://github.com/apache/airflow/issues/16148#issuecomment-855240855


   Thanks for being so persistent @ConstantinoSchillebeeckx . Great investigation! It's a very subtle problem it seems.
   
   I believe @mik-laj is right. We seem to have indeed a problem here introduced by #15599. In SecretMasker all "iterable" objects are converted into lists by the SecretMasker https://github.com/apache/airflow/pull/15599/files#diff-63a47f524bd31519c96c4f7376b864a77eedf8d6cf59ef6e5c86d575d53a74c6R221 . Unfortunately, the StreamingBody is iterable: https://github.com/boto/botocore/blob/87facf71d42fa6a17d22ff946d76d3ea1d01f561/botocore/response.py#L89 - and it will read all the content of the body while converting it. The original Debug statement works fine without the SecretMasker because it uses %s to format the object, which will not call __iter()__ method (it will just print the object + address).
   
   This would only show itself if (like you) you want to debug botocore and you set the level of logger of botocore to "Debug". That's wny seting the "botocore.parser" level to "Critical" fixed the issue.
   
   I think there is no easy way to avoid similar problems and we should remove the "iterable" handling and just limit it to cases where those are explicit 'lists' of objects rather than Iterable objects ? Simply adding list to (set, tuple) above should do the job..
   
   @ashb @ephraimbuddy  - WDYT?  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org