[jira] [Commented] (AIRFLOW-3723) Add Gzip capability to existing mongo_to_S3 operator

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258086#comment-17258086 ] 

ASF GitHub Bot commented on AIRFLOW-3723:
-----------------------------------------

feluelle commented on a change in pull request #13187:
URL: https://github.com/apache/airflow/pull/13187#discussion_r551193283



##########
File path: airflow/providers/amazon/aws/hooks/s3.py
##########
@@ -539,11 +540,24 @@ def load_string(
         :param acl_policy: The string to specify the canned ACL policy for the
             object to be uploaded
         :type acl_policy: str
+        :param compression: Type of compression to use, currently only gzip is supported.
+        :type compression: str
         """
         encoding = encoding or 'utf-8'
 
         bytes_data = string_data.encode(encoding)
+
+        # Compress string
+        if compression not in (None, 'gzip'):
+            raise NotImplementedError(
+                "Received {} compression type. String "
+                "can currently be compressed in gzip only.".format(compression)
+            )

Review comment:
       Maybe we could automatically create the error message based on available compressions if we store these in a variable and read from it.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Add Gzip capability to existing mongo_to_S3 operator
> ----------------------------------------------------
>
>                 Key: AIRFLOW-3723
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3723
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.1
>            Reporter: deepak saroha
>            Assignee: deepak saroha
>            Priority: Minor
>             Fix For: 2.0.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> A feature to add gzip capabilities while writing to S3 from Mongo is crucial as S3 has space base billing system. Gzipping it will greatly benefit.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)