You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/07 16:17:43 UTC
[GitHub] [airflow] JavierLopezT opened a new pull request #11332: SnowflakeToS3Operator
JavierLopezT opened a new pull request #11332:
URL: https://github.com/apache/airflow/pull/11332
New transfer operator, Snowflake To S3
I still have to include tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
feluelle commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-771009993
@JavierLopezT The static check is failing because of:
```
Validate providers.yaml files............................................................Failed
- hook id: provider-yamls
- exit code: 1
Checking integration duplicates
Checking completeness of list of {sensors, hooks, operators}
Checking for duplicates in list of {sensors, hooks, operators}
Checking completeness of list of transfers
Checking for duplicates in list of transfers
Checking connection classes belong to package
Checking doc files
Detect unregistered integrations
Found 2 errors
The `airflow.providers.aws.transfers.snowflake_to_s3` object in transfers list in airflow/providers/amazon/provider.yaml does not start with the expected airflow.providers.amazon.
Incorrect content of key 'transfers/python-module' in file: airflow/providers/amazon/provider.yaml
Items in the first set but not the second:
'airflow.providers.amazon.aws.transfers.snowflake_to_s3'
Items in the second set but not the first:
'airflow.providers.aws.transfers.snowflake_to_s3'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-720435126
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r502935092
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,82 @@
+"""
Review comment:
Missing Apache license header
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r515930429
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
Review comment:
I have added some docs as well as another possibility to unload. What do you think?
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like ``type = csv field_delimiter = ',' skip_header = 1``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
Review comment:
Done it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r513250323
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like ``type = csv field_delimiter = ',' skip_header = 1``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[list] = None,
+ autocommit: bool = True,
+ include_header: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.file_format = file_format
+ self.snowflake_conn_id = snowflake_conn_id
+ self.schema = schema
+ self.table = table
+ self.sql = sql
+ self.unload_options = unload_options or [] # type: List
+ self.autocommit = autocommit
+ self.include_header = include_header
+
+ def execute(self, context):
+ snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ unload_options = '\n\t'.join(self.unload_options)
+
+ if not self.sql:
+ self.sql = self.schema + '.' + self.table
Review comment:
```suggestion
```
I think this makes it unnecessary "complex".
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like ``type = csv field_delimiter = ',' skip_header = 1``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
Review comment:
```suggestion
table_or_query: Optional[str] = None,
```
I would suggest to keep it rather simple and allow passing table or query to one variable and just pass it to the `COPY` string. Otherwise you would have to do more validation on this - so that either schema and table or a query was passed.
Ref: https://docs.snowflake.com/en/sql-reference/sql/copy-into-location.html#syntax
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r506642714
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[list] = None,
+ autocommit: bool = True,
+ include_header: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.file_format = file_format
+ self.snowflake_conn_id = snowflake_conn_id
+ self.schema = schema
+ self.table = table
+ self.sql = sql
+ self.unload_options = unload_options or [] # type: List
+ self.autocommit = autocommit
+ self.include_header = include_header
+
+ def execute(self, context):
+ snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ unload_options = '\n\t\t\t'.join(self.unload_options)
+
+ if not self.sql:
+ self.sql = self.schema + '.' + self.table
+
+ unload_query = f"""
+ COPY INTO 's3://{self.s3_bucket}/{self.s3_key}'
+ FROM ({self.sql})
+ STORAGE_INTEGRATION = S3
+ FILE_FORMAT = ({self.file_format})
+ {unload_options}
+ HEADER = {self.include_header};
+ """
Review comment:
Do we need this `\n\t\t\t` because the `unload_query` is indented? Should we be able to dedent it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT closed pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT closed pull request #11332:
URL: https://github.com/apache/airflow/pull/11332
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r542127785
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+from airflow.utils.decorators import apply_defaults
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3. It is not necessary a conn to S3 because
+ Snowflake handles it. You have to follow one of the setups in:
+ https://docs.snowflake.com/en/user-guide/data-load-s3-config.html
+
+ :param stage: Copy the data into a Snowflake Stage. This allows you to use the method
+ 'Configuring AWS IAM User Credentials' for unloading data to s3. If this is true,
+ s3_bucket and file_format are not necessaries.
+ :type stage: bool
+ :param warehouse: reference to a specific snowflake warehouse to override the one in conn
+ :type warehouse: str
+ :param database: reference to a specific snowflake database to override the one in conn
+ :type warehouse: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved. For using it, you
+ should have done the one-time setup 'Configuring a Snowflake Storage Integration'
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake
+ or hardcoded one like ``type = csv field_delimiter = ',' skip_header = 1``
+ :type file_format: str
+ :param query_or_table: query or full table to unload. If table, it must include the schema like
+ ``schema.table`
Review comment:
```suggestion
`schema.table`
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-732104490
> @JavierLopezT the checks are failing with
> `tests/providers/amazon/aws/transfers/test_snowflake_to_s3.py:43:8: E1125: Missing mandatory keyword argument 'query_or_table' in constructor call (missing-kwoa)`
Thanks @RosterIn . That is because I didn't update the code in the tests to match the operator, because as I said in https://github.com/apache/airflow/pull/11332#issuecomment-720435299 I wanted some feedback before proceeding. Probably my mistake to not tag anyone. If you or @feluelle don't have any objections, I'll fix the tests ASAP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-732104490
> @JavierLopezT the checks are failing with
> `tests/providers/amazon/aws/transfers/test_snowflake_to_s3.py:43:8: E1125: Missing mandatory keyword argument 'query_or_table' in constructor call (missing-kwoa)`
Thanks @RosterIn . That is because I didn't update the code in the tests to match the operator, because as I said in https://github.com/apache/airflow/pull/11332#issuecomment-720435299 I wanted some feedback before proceeding. Probably my mistake, not tagging anyone. If you or @feluelle don't have any objections, I'll fix the tests ASAP
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-714292985
[The Workflow run](https://github.com/apache/airflow/actions/runs/321499965) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-709449318
Hello. I get the error
```
Warning, treated as error:
/opt/airflow/docs/_api/airflow/providers/amazon/aws/transfers/snowflake_to_s3/index.rst:30:Unexpected indentation.
==================== Error 1 ====================
Sphinx spellcheck returned non-zero exit status: 2.
```
I don't know what is this about. Could you help me, please? Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-771726104
[The Workflow run](https://github.com/apache/airflow/actions/runs/530729330) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-714415353
[The Workflow run](https://github.com/apache/airflow/actions/runs/321870976) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-713820037
[The Workflow run](https://github.com/apache/airflow/actions/runs/320516434) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-721446212
Can you please rebase your PR on latest Master since we have applied [Black](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) and [PyUpgrade](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) on Master.
It will help if your squash your commits into single commit first so that there are less conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-714293335
[The Workflow run](https://github.com/apache/airflow/actions/runs/321501833) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r545753211
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+from airflow.utils.decorators import apply_defaults
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3. It is not necessary a conn to S3 because
+ Snowflake handles it. You have to follow one of the setups in:
+ https://docs.snowflake.com/en/user-guide/data-load-s3-config.html
+
+ :param stage: Copy the data into a Snowflake Stage. This allows you to use the method
+ 'Configuring AWS IAM User Credentials' for unloading data to s3. If this is true,
+ s3_bucket and file_format are not necessaries.
+ :type stage: bool
Review comment:
Actually this is a mistake, stage is a string. I have already changed it. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ryw commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
ryw commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-717491372
@turbaszek is this mergeable or waiting on another review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] feluelle commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
feluelle commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r545725384
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+from airflow.utils.decorators import apply_defaults
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3. It is not necessary a conn to S3 because
+ Snowflake handles it. You have to follow one of the setups in:
+ https://docs.snowflake.com/en/user-guide/data-load-s3-config.html
+
+ :param stage: Copy the data into a Snowflake Stage. This allows you to use the method
+ 'Configuring AWS IAM User Credentials' for unloading data to s3. If this is true,
+ s3_bucket and file_format are not necessaries.
+ :type stage: bool
+ :param warehouse: reference to a specific snowflake warehouse to override the one in conn
+ :type warehouse: str
+ :param database: reference to a specific snowflake database to override the one in conn
+ :type warehouse: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved. For using it, you
+ should have done the one-time setup 'Configuring a Snowflake Storage Integration'
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake
+ or hardcoded one like ``type = csv field_delimiter = ',' skip_header = 1``
+ :type file_format: str
+ :param query_or_table: query or full table to unload. If table, it must include the schema like
+ `schema.table`
+ :type query_or_table: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str`
Review comment:
```suggestion
:type snowflake_conn_id: str
```
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,150 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+
+from airflow.models import BaseOperator
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+from airflow.utils.decorators import apply_defaults
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3. It is not necessary a conn to S3 because
+ Snowflake handles it. You have to follow one of the setups in:
+ https://docs.snowflake.com/en/user-guide/data-load-s3-config.html
+
+ :param stage: Copy the data into a Snowflake Stage. This allows you to use the method
+ 'Configuring AWS IAM User Credentials' for unloading data to s3. If this is true,
+ s3_bucket and file_format are not necessaries.
+ :type stage: bool
Review comment:
```suggestion
:param use_stage: Copy the data into a Snowflake Stage. This allows you to use the method
'Configuring AWS IAM User Credentials' for unloading data to s3. If this is true,
s3_bucket and file_format are not necessaries.
:type use_stage: bool
```
WDYT? If yes, then obviously you have to change the references. (I do not comment on them. :))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r509312388
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[list] = None,
+ autocommit: bool = True,
+ include_header: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.file_format = file_format
+ self.snowflake_conn_id = snowflake_conn_id
+ self.schema = schema
+ self.table = table
+ self.sql = sql
+ self.unload_options = unload_options or [] # type: List
+ self.autocommit = autocommit
+ self.include_header = include_header
+
+ def execute(self, context):
+ snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ unload_options = '\n\t\t\t'.join(self.unload_options)
+
+ if not self.sql:
+ self.sql = self.schema + '.' + self.table
+
+ unload_query = f"""
+ COPY INTO 's3://{self.s3_bucket}/{self.s3_key}'
+ FROM ({self.sql})
+ STORAGE_INTEGRATION = S3
+ FILE_FORMAT = ({self.file_format})
+ {unload_options}
+ HEADER = {self.include_header};
+ """
Review comment:
I think if we can improve the code and remove this strange string then it may be worth doing it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT edited a comment on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT edited a comment on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-720435299
I have added a new possibility to make the unload. Not sure if this is the more elegant way to do it, though. So before applying the changes in the tests I would like some feedback. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-731711852
@JavierLopezT the checks are failing with
`tests/providers/amazon/aws/transfers/test_snowflake_to_s3.py:43:8: E1125: Missing mandatory keyword argument 'query_or_table' in constructor call (missing-kwoa)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-721655309
> Can you please rebase your PR on latest Master since we have applied [Black](https://github.com/apache/airflow/commit/4e8f9cc8d02b29c325b8a5a76b4837671bdf5f68) and [PyUpgrade](https://github.com/apache/airflow/commit/8c42cf1b00c90f0d7f11b8a3a455381de8e003c5) on Master.
>
> It will help if your squash your commits into single commit first so that there are less conflicts.
I have done it (without squashing though, I didn't know how to do it), but has it worked? I didn't see any conflict
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r506641708
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
Review comment:
Missing indent
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
Review comment:
Missing indent
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-705418532
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/295063117) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-705418532
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/295063117) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-763989313
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-721692003
[The Workflow run](https://github.com/apache/airflow/actions/runs/345470604) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r502935265
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,82 @@
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional, Union
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a str of UNLOAD options. For instance, a valid string would be:
+ ``SINGLE=TRUE
+ MAX_FILE_SIZE = 5368709119
+ OVERWRITE = TRUE``
+ :type unload_options: str
+ """
+
+ template_fields = ('s3_key', 's3_bucket', 'sql', 'table', 'snowflake_conn_id',)
+ template_ext = ('.sql',)
+ ui_color = '#ededed'
+
+ @apply_defaults
+ def __init__(
+ self,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[str] = None,
+ autocommit: bool = True,
+ *args, **kwargs) -> None:
Review comment:
```suggestion
self,
*,
s3_bucket: str,
s3_key: str,
file_format: str,
schema: Optional[str] = None,
table: Optional[str] = None,
sql: Optional[str] = None,
snowflake_conn_id: str = "snowflake_default",
unload_options: Optional[str] = None,
autocommit: bool = True,
**kwargs) -> None:
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-721657685
[The Workflow run](https://github.com/apache/airflow/actions/runs/345359586) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
RosterIn commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r513005638
##########
File path: tests/providers/amazon/aws/transfers/test_snowflake_to_s3.py
##########
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+import unittest
+from unittest import mock
+
+from airflow.providers.amazon.aws.transfers.snowflake_to_s3 import SnowflakeToS3Operator
+from tests.test_utils.asserts import assert_equal_ignore_multiple_spaces
+
+
+class TestSnowflakeToS3Transfer(unittest.TestCase):
+ @mock.patch("airflow.providers.snowflake.hooks.snowflake.SnowflakeHook.run")
+ def test_execute(
+ self,
+ mock_run,
+ ):
+ schema = "schema"
+ table = "table"
+ s3_bucket = "bucket"
+ s3_key = "key"
+ unload_options = [
+ 'OVERWRITE = TRUE',
+ ]
+ file_format = "file_format"
+ sql = None
Review comment:
The operator has two modes one when sql param is provided and one when it's not. Shouldn't be two test cases?
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,105 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Transfers data from Snowflake into a S3 Bucket."""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
Review comment:
I think this operator needs further documentation.
Some users might be surprised that there is no `aws_conn` here and the `aws_access_key_id`, `aws_secret_access_key` are defined in snowflake connection. I think this is not trivial.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-763989313
[The Workflow run](https://github.com/apache/airflow/actions/runs/499702046) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-714441417
[The Workflow run](https://github.com/apache/airflow/actions/runs/321952645) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-705136790
[The Build Workflow run](https://github.com/apache/airflow/actions/runs/294039474) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-713819701
[The Workflow run](https://github.com/apache/airflow/actions/runs/320500517) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-720435299
I have added a new possibility to make the unload. Not sure if this is the more elegant way to do it, though. So before applying the changes in the test I would like some feedback. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r510177678
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[list] = None,
+ autocommit: bool = True,
+ include_header: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.file_format = file_format
+ self.snowflake_conn_id = snowflake_conn_id
+ self.schema = schema
+ self.table = table
+ self.sql = sql
+ self.unload_options = unload_options or [] # type: List
+ self.autocommit = autocommit
+ self.include_header = include_header
+
+ def execute(self, context):
+ snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ unload_options = '\n\t\t\t'.join(self.unload_options)
+
+ if not self.sql:
+ self.sql = self.schema + '.' + self.table
+
+ unload_query = f"""
+ COPY INTO 's3://{self.s3_bucket}/{self.s3_key}'
+ FROM ({self.sql})
+ STORAGE_INTEGRATION = S3
+ FILE_FORMAT = ({self.file_format})
+ {unload_options}
+ HEADER = {self.include_header};
+ """
Review comment:
I was able to remove two \t characters. Otherwise, the query in the log is unintended.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-709495719
[The Workflow run](https://github.com/apache/airflow/actions/runs/309103523) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#discussion_r509294036
##########
File path: airflow/providers/amazon/aws/transfers/snowflake_to_s3.py
##########
@@ -0,0 +1,111 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""
+Transfers data from Snowflake into a S3 Bucket.
+"""
+from typing import List, Optional
+from airflow.models import BaseOperator
+from airflow.utils.decorators import apply_defaults
+from airflow.providers.snowflake.hooks.snowflake import SnowflakeHook
+
+
+class SnowflakeToS3Operator(BaseOperator):
+ """
+ UNLOAD a query or a table from Snowflake to S3
+
+ :param schema: reference to a specific schema in snowflake database
+ :type schema: str
+ :param table: reference to a specific table in snowflake database
+ :type table: str
+ :param s3_bucket: reference to a specific S3 bucket where the data will be saved
+ :type s3_bucket: str
+ :param s3_key: reference to a specific S3 key within the previous bucket
+ :type s3_key: str
+ :param file_format: can be either a previous file format created in Snowflake console
+ or hardcoded one like
+ ``type = csv
+ field_delimiter = ','
+ skip_header = 1
+ compression=None``
+ :type file_format: str
+ :param sql: optional parameter to unload a customized query instead an entire table
+ :type sql: str
+ :param snowflake_conn_id: reference to a specific snowflake database
+ :type snowflake_conn_id: str
+ :param unload_options: reference to a list of UNLOAD options (SINGLE, MAX_FILE_SIZE,
+ OVERWRITE etc). Each element of the list has to be a string
+ :type unload_options: list
+ """
+
+ template_fields = (
+ 's3_key',
+ 's3_bucket',
+ 'sql',
+ 'table',
+ 'snowflake_conn_id',
+ )
+ template_ext = ('.sql',)
+ ui_color = '#ffebb2'
+
+ @apply_defaults
+ def __init__(
+ self,
+ *,
+ s3_bucket: str,
+ s3_key: str,
+ file_format: str,
+ schema: Optional[str] = None,
+ table: Optional[str] = None,
+ sql: Optional[str] = None,
+ snowflake_conn_id: str = "snowflake_default",
+ unload_options: Optional[list] = None,
+ autocommit: bool = True,
+ include_header: bool = False,
+ **kwargs,
+ ) -> None:
+ super().__init__(**kwargs)
+ self.s3_bucket = s3_bucket
+ self.s3_key = s3_key
+ self.file_format = file_format
+ self.snowflake_conn_id = snowflake_conn_id
+ self.schema = schema
+ self.table = table
+ self.sql = sql
+ self.unload_options = unload_options or [] # type: List
+ self.autocommit = autocommit
+ self.include_header = include_header
+
+ def execute(self, context):
+ snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ unload_options = '\n\t\t\t'.join(self.unload_options)
+
+ if not self.sql:
+ self.sql = self.schema + '.' + self.table
+
+ unload_query = f"""
+ COPY INTO 's3://{self.s3_bucket}/{self.s3_key}'
+ FROM ({self.sql})
+ STORAGE_INTEGRATION = S3
+ FILE_FORMAT = ({self.file_format})
+ {unload_options}
+ HEADER = {self.include_header};
+ """
Review comment:
To be honest, I just copied it from the existing RedshiftToS3Operator. I don't know if we should maintain this or modifying. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-763989972
[The Workflow run](https://github.com/apache/airflow/actions/runs/499652455) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-721692256
[The Workflow run](https://github.com/apache/airflow/actions/runs/345471391) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-714441771
[The Workflow run](https://github.com/apache/airflow/actions/runs/321959801) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-709389941
[The Workflow run](https://github.com/apache/airflow/actions/runs/308844700) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-734308737
[The Workflow run](https://github.com/apache/airflow/actions/runs/385337295) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-771726104
[The Workflow run](https://github.com/apache/airflow/actions/runs/530729330) is cancelling this PR. Building images for the PR has failed. Follow the the workflow link to check the reason.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11332: SnowflakeToS3Operator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11332:
URL: https://github.com/apache/airflow/pull/11332#issuecomment-709356989
[The Workflow run](https://github.com/apache/airflow/actions/runs/308755385) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org