You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "jacobcbeaudin (via GitHub)" <gi...@apache.org> on 2026/02/14 14:05:15 UTC
[I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
jacobcbeaudin opened a new issue, #40167:
URL: https://github.com/apache/airflow/issues/40167
### Description
Enable scheduling and execution of Snowflake Notebooks from Apache Airflow. This feature would allow users to orchestrate and manage Snowflake Notebooks directly within Airflow, leveraging Airflow's scheduling, dependency management, and monitoring capabilities.
### Use case/motivation
The recent launch of Snowflake Notebooks offers a powerful environment for data engineering, analytics, and machine learning workflows. By integrating Snowflake Notebooks with Airflow, users can automate their data workflows more efficiently. This integration will enable users to:
- Schedule regular execution of Snowflake Notebooks.
- Incorporate Snowflake Notebooks into larger, multi-step data pipelines.
- Use Airflow's monitoring and alerting features to manage the execution of Snowflake Notebooks.
- Reduce the need for manual intervention, thereby increasing productivity and ensuring consistency in data processes.
This feature will greatly benefit data teams who already use Snowflake and Airflow, providing a seamless and automated way to manage their end-to-end data workflows.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2160790471
@jacobcbeaudin
Just to be clear, does not airflow [SnowflakeSqlApiOperator](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/operators/snowflake.html#snowflakesqlapioperator) sufficient with [execute notebook sql](https://docs.snowflake.com/en/sql-reference/sql/execute-notebook) statement?
```SQL
EXECUTE NOTEBOOK notebook_name();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2160501950
@Lee-W please assign it in my name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "jacobcbeaudin (via GitHub)" <gi...@apache.org>.
jacobcbeaudin commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2163758920
> @jacobcbeaudin Just to be clear, does not airflow [SnowflakeSqlApiOperator](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/operators/snowflake.html#snowflakesqlapioperator) sufficient with [execute notebook sql](https://docs.snowflake.com/en/sql-reference/sql/execute-notebook) statement?
>
> ```sql
> EXECUTE NOTEBOOK notebook_name();
> ```
Thank you for the reference. The **`SnowflakeSqlApiOperator`** with the **`EXECUTE NOTEBOOK`** statement is indeed a sufficient current solution. However, I anticipate Snowflake will soon offer options to specify parameters for notebooks. At that point, the SQL commands may become less user-friendly. My feature request aims to enhance user experience similar to Databricks jobs Airflow operators, which provide a more streamlined and intuitive interface for managing complex workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "jacobcbeaudin (via GitHub)" <gi...@apache.org>.
jacobcbeaudin commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-4049069668
I've opened a PR for this: #63470
The implementation adds a dedicated `SnowflakeNotebookOperator` that:
- Submits `EXECUTE NOTEBOOK` via the Snowflake SQL API
- Polls for completion and pushes the query ID to XCom
- Supports deferrable mode from the start, reusing the existing `SnowflakeSqlApiTrigger`
- Supports passing parameters to notebooks
No new hook or trigger code was needed. The operator reuses `SnowflakeSqlApiHook` and `SnowflakeSqlApiTrigger`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-3986486343
@blcksrx We are unassigning you from this issue as part of our updated [assignment policy](https://github.com/apache/airflow/blob/main/contributing-docs/04_how_to_contribute.rst#contribute-code-changes).
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2159291981
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2165110238
@jacobcbeaudin @Lee-W
I understand, let's close this issue then, once Snowflake announced the official documentation then we'll continue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "blcksrx (via GitHub)" <gi...@apache.org>.
blcksrx commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-3773416837
@potiuk Im going to work on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #40167: Integrate Snowflake Notebook Scheduling with Apache Airflow
URL: https://github.com/apache/airflow/issues/40167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W closed issue #40167: Integrate Snowflake Notebook Scheduling with Apache Airflow
URL: https://github.com/apache/airflow/issues/40167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "saurabhkpdev (via GitHub)" <gi...@apache.org>.
saurabhkpdev commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-3670267029
This would be great feature . All the different shells of a notebook turning into airflow task and forming a DAG. Very similar to what cosmos does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-3671357555
> This would be great feature . All the different shells of a notebook turning into airflow task and forming a DAG. Very similar to what cosmos does.
Feel free to contribute it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "jacobcbeaudin (via GitHub)" <gi...@apache.org>.
jacobcbeaudin commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-4049025785
I'm going to implement this. Will add a dedicated `SnowflakeNotebookOperator` backed by the existing `SnowflakeSqlApiHook` and `SnowflakeSqlApiTrigger`, with deferrable mode supported from the start — as suggested in the comment above. PR coming soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "jacobcbeaudin (via GitHub)" <gi...@apache.org>.
jacobcbeaudin commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2163759566
> @jacobcbeaudin Just to be clear, does not airflow [SnowflakeSqlApiOperator](https://airflow.apache.org/docs/apache-airflow-providers-snowflake/stable/operators/snowflake.html#snowflakesqlapioperator) sufficient with [execute notebook sql](https://docs.snowflake.com/en/sql-reference/sql/execute-notebook) statement?
>
> ```sql
> EXECUTE NOTEBOOK notebook_name();
> ```
@blcksrx Thank you for the reference. The **`SnowflakeSqlApiOperator`** with the **`EXECUTE NOTEBOOK`** statement is indeed a sufficient current solution. However, I anticipate Snowflake will soon offer options to specify parameters for notebooks. At that point, the SQL commands may become less user-friendly. My feature request aims to enhance user experience similar to Databricks jobs Airflow operators, which provide a more streamlined and intuitive interface for managing complex workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-2165144309
Sounds good to me 👍 @jacobcbeaudin I'll close it for now. But please feel free to reopen it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "passionworkeer (via GitHub)" <gi...@apache.org>.
passionworkeer commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-4045470532
A clean provider design might be: add a dedicated SnowflakeNotebookOperator backed by a small hook method that (1) submits a notebook execution job, (2) polls status until terminal state, and (3) returns the execution/job id via XCom. That keeps scheduling semantics in the operator while leaving API details in the hook. It would also be useful to align with other long-running provider operators by supporting deferrable mode from the start, since notebook execution can be much longer than a typical SQL call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Integrate Snowflake Notebook Scheduling with Apache Airflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #40167:
URL: https://github.com/apache/airflow/issues/40167#issuecomment-3901972511
Feel free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org