You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/23 15:40:59 UTC
[GitHub] [airflow] JavierLopezT opened a new pull request #12564: Add connection arguments in S3ToSnowflakeOperator
JavierLopezT opened a new pull request #12564:
URL: https://github.com/apache/airflow/pull/12564
Right now in this operator, if you want to use several warehouses, you have to create different connections, which is not ideal. With this change, you can specify the warehouse in each task, as well as other things related to the connection: role, authenticator, and session parameters.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #12564:
URL: https://github.com/apache/airflow/pull/12564#discussion_r528813767
##########
File path: airflow/providers/snowflake/transfers/s3_to_snowflake.py
##########
@@ -58,22 +77,38 @@ def __init__(
file_format: str,
schema: str, # TODO: shouldn't be required, rely on session/user defaults
columns_array: Optional[list] = None,
+ warehouse: Optional[str] = None,
autocommit: bool = True,
snowflake_conn_id: str = 'snowflake_default',
+ role: Optional[str] = None,
+ authenticator: Optional[str] = None,
+ session_parameters: Optional[dict] = None,
**kwargs,
) -> None:
super().__init__(**kwargs)
self.s3_keys = s3_keys
self.table = table
+ self.warehouse = warehouse
self.stage = stage
self.file_format = file_format
self.schema = schema
self.columns_array = columns_array
self.autocommit = autocommit
self.snowflake_conn_id = snowflake_conn_id
+ self.role = role
+ self.authenticator = authenticator
+ self.session_parameters = session_parameters
def execute(self, context: Any) -> None:
- snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ snowflake_hook = SnowflakeHook(
+ snowflake_conn_id=self.snowflake_conn_id,
+ warehouse=self.warehouse,
+ database=self.database,
Review comment:
Where do you set the value of `self.database` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12564:
URL: https://github.com/apache/airflow/pull/12564#issuecomment-761613874
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest master or amend the last commit of the PR, and push it with --force-with-lease.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] JavierLopezT commented on a change in pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
JavierLopezT commented on a change in pull request #12564:
URL: https://github.com/apache/airflow/pull/12564#discussion_r528817761
##########
File path: airflow/providers/snowflake/transfers/s3_to_snowflake.py
##########
@@ -58,22 +77,38 @@ def __init__(
file_format: str,
schema: str, # TODO: shouldn't be required, rely on session/user defaults
columns_array: Optional[list] = None,
+ warehouse: Optional[str] = None,
autocommit: bool = True,
snowflake_conn_id: str = 'snowflake_default',
+ role: Optional[str] = None,
+ authenticator: Optional[str] = None,
+ session_parameters: Optional[dict] = None,
**kwargs,
) -> None:
super().__init__(**kwargs)
self.s3_keys = s3_keys
self.table = table
+ self.warehouse = warehouse
self.stage = stage
self.file_format = file_format
self.schema = schema
self.columns_array = columns_array
self.autocommit = autocommit
self.snowflake_conn_id = snowflake_conn_id
+ self.role = role
+ self.authenticator = authenticator
+ self.session_parameters = session_parameters
def execute(self, context: Any) -> None:
- snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ snowflake_hook = SnowflakeHook(
+ snowflake_conn_id=self.snowflake_conn_id,
+ warehouse=self.warehouse,
+ database=self.database,
Review comment:
My mistake. I think in this case is not necessary, so I'll delete it from here
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a change in pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
eladkal commented on a change in pull request #12564:
URL: https://github.com/apache/airflow/pull/12564#discussion_r528819179
##########
File path: airflow/providers/snowflake/transfers/s3_to_snowflake.py
##########
@@ -58,22 +77,38 @@ def __init__(
file_format: str,
schema: str, # TODO: shouldn't be required, rely on session/user defaults
columns_array: Optional[list] = None,
+ warehouse: Optional[str] = None,
autocommit: bool = True,
snowflake_conn_id: str = 'snowflake_default',
+ role: Optional[str] = None,
+ authenticator: Optional[str] = None,
+ session_parameters: Optional[dict] = None,
**kwargs,
) -> None:
super().__init__(**kwargs)
self.s3_keys = s3_keys
self.table = table
+ self.warehouse = warehouse
self.stage = stage
self.file_format = file_format
self.schema = schema
self.columns_array = columns_array
self.autocommit = autocommit
self.snowflake_conn_id = snowflake_conn_id
+ self.role = role
+ self.authenticator = authenticator
+ self.session_parameters = session_parameters
def execute(self, context: Any) -> None:
- snowflake_hook = SnowflakeHook(snowflake_conn_id=self.snowflake_conn_id)
+ snowflake_hook = SnowflakeHook(
+ snowflake_conn_id=self.snowflake_conn_id,
+ warehouse=self.warehouse,
+ database=self.database,
Review comment:
Why? like schema you can also select different DBs.
Do you see value in limiting it only to the DB defined in the connection?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] milton0825 merged pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
milton0825 merged pull request #12564:
URL: https://github.com/apache/airflow/pull/12564
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12564: Add connection arguments in S3ToSnowflakeOperator
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12564:
URL: https://github.com/apache/airflow/pull/12564#issuecomment-732284444
[The Workflow run](https://github.com/apache/airflow/actions/runs/379365341) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org