You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "frodo2000 (via GitHub)" <gi...@apache.org> on 2023/12/18 11:20:20 UTC
[I] Parameters are not templated for Snowflake Operator [airflow]
frodo2000 opened a new issue, #36277:
URL: https://github.com/apache/airflow/issues/36277
### Apache Airflow version
2.7.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Due to lack of "parameters" value in template_field variable - there is no way to use templated value (ex operator.output)
### What you think should happen instead?
Snowflake Operator should work as same as SQLExecuteQueryOperator
### How to reproduce
`import airflow
from airflow import DAG
from airflow.models import Variable
from airflow.utils.task_group import TaskGroup
from airflow.operators.dummy import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
from dateutil import parser
from datetime import datetime, timedelta, date, time
import pendulum
local_tz = pendulum.timezone("Europe/Warsaw")
default_args = {
'depends_on_past': False,
'retries': 1,
'provide_context': True,
}
with DAG(
dag_id='dag_test_snowflake_params',
schedule_interval=None,
max_active_runs=1,
catchup=True,
dagrun_timeout=timedelta(minutes=420),
start_date=pendulum.datetime(2023, 1, 3, tz=local_tz),
default_args=default_args
) as dag:
start_task = DummyOperator(
task_id='start_task'
)
def test_function():
return 10
python_task = PythonOperator(
task_id='python_task',
python_callable=test_function
)
snowflake_task = SnowflakeOperator(
task_id='snowflake_task',
sql='select %(test_param)s',
snowflake_conn_id='snowflake_conn',
parameters={
"test_param":"{}".format(python_task.output)
}
)
sqlexecutequery_task = SQLExecuteQueryOperator(
task_id='sqlexecutequery_task',
sql='select %(test_param)s',
conn_id='snowflake_conn',
parameters={
"test_param":"{}".format(python_task.output)
}
)
start_task >> python_task >> snowflake_task >> sqlexecutequery_task
`
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
Tested on snowflake-provider 5.0.1 and 5.2.0 versions
### Deployment
Virtualenv installation
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameters are not templated for Snowflake Operator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #36277: Parameters are not templated for Snowflake Operator
URL: https://github.com/apache/airflow/issues/36277
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameters are not templated for Snowflake Operator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #36277:
URL: https://github.com/apache/airflow/issues/36277#issuecomment-1860299086
Thank you for stating watch on this but for the moment I see no action item here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameters are not templated for Snowflake Operator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #36277:
URL: https://github.com/apache/airflow/issues/36277#issuecomment-1860294443
> Nevertheless if you will not plan remove SnowflakeOperator in the next release it should be also fixed - even if it is flaged as deprecated.
We do plan to remove it in the following major release (whenever it will be).
We try to avoid creating major release just to remove stuff release should have user value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameters are not templated for Snowflake Operator [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #36277:
URL: https://github.com/apache/airflow/issues/36277#issuecomment-1860240378
> Snowflake Operator should work as same as SQLExecuteQueryOperator
Please clarify what is the problem with `SQLExecuteQueryOperator`.
`SnowflakeOperator` is deprecated.
If what you are after works with `SQLExecuteQueryOperator` then there is nothing to fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Parameters are not templated for Snowflake Operator [airflow]
Posted by "frodo2000 (via GitHub)" <gi...@apache.org>.
frodo2000 commented on issue #36277:
URL: https://github.com/apache/airflow/issues/36277#issuecomment-1860245369
SQLExecuteQueryOperator works correctly. Nevertheless if you will not plan remove SnowflakeOperator in the next release it should be also fixed - even if it is flaged as deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org