You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/08 16:08:37 UTC
[GitHub] [airflow] Bowrna opened a new pull request #19470: Mssql dag
Bowrna opened a new pull request #19470:
URL: https://github.com/apache/airflow/pull/19470
MSSQL Example DAG. I have appended the working code by testing in SQL server.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19470: Mssql dag
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-963359126
Cool. Looks good, I would like however to add one more thing (this might be a separate PR but it would be great to add it here).
The comments `[START howto_operator_mssql]` and the like are used to generate HowTo documentation out of the examples. This is cool the example DAG works, but the biggest benefit of it is the documentaiton on using the operators extracts the snippets of that working example dags and puts them in "Howto" guides. For example see how it is done in MySQL: https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-mysql/operators.rst
It would be great to heave such HowTo document added. Can you add it please too ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Mssql dag
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964084538
@potiuk I have added the docs and i tried to commit the doc into repository and the pre-commit build fails
```Validate providers.yaml files............................................................Failed
- hook id: provider-yamls
- exit code: 1
Checking integration duplicates
Checking completeness of list of {sensors, hooks, operators}
-- {sensors, hooks, operators} - Expected modules(Left): Current Modules(Right)
Checking for duplicates in list of {sensors, hooks, operators}
Checking completeness of list of transfers
-- Expected transfers modules(Left): Current transfers Modules(Right)
Checking for duplicates in list of transfers
Checking connection classes belong to package
-- Checking providers: present in code(left), mentioned in /Users/sathishkannan/code/airflow/.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml (right)
Checking doc files
-- Checking document urls: expected(left), current(right)
-- Items in the left set but not the right:
'/docs/apache-airflow-providers-microsoft-mssql/operators.rst'
```
I could not understand this failure message. Can you help me to understand this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-968815085
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964848616
thank you @eladkal and @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on a change in pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#discussion_r748846716
##########
File path: docs/apache-airflow-providers-microsoft-mssql/operators.rst
##########
@@ -0,0 +1,112 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+How-to Guide for MsSqlOperator
+=================================
+
+.. _howto/operator:MsSqlOperator:
+
+Introduction
+------------
+The purpose of MSSQL Operator is to define tasks involving interactions with the MSSQL database.
+
+Use the :class:`~airflow.providers.microsoft.mssql.operators.mssql.MsSqlOperator` to execute
+SQL commands in MSSQL database.
+
+Common Database Operations with MsSqlOperator
+------------------------------------------------
+
+To use the mssql operator to carry out SQL request, two parameters are required: ``sql`` and ``mssql_conn_id``.
+These two parameters are eventually fed to the MSSQL hook object that interacts directly with the MSSQL database.
+
+Creating a MSSQL database table
+----------------------------------
+
+The code snippets below are based on Airflow-2.2
+
+An example usage of the MsSqlOperator is as follows:
+
+.. exampleinclude:: /../../airflow/providers/microsoft/mssql/example_dags/example_mssql.py
+ :language: python
+ :start-after: [START howto_operator_mssql]
+ :end-before: [END howto_operator_mssql]
+
+You can also use an external file to execute the SQL commands. Script folder must be at the same level as DAG.py file.
+This way you can easily maintain the SQL queries separated from the code.
+
+.. exampleinclude:: /../../airflow/providers/microsoft/mssql/example_dags/example_mssql.py
+ :language: python
+ :start-after: [START mssql_operator_howto_guide_create_table_mssql_from_external_file]
+ :end-before: [END mssql_operator_howto_guide_create_table_mssql_from_external_file]
+
+
+Your ``dags/create_table.sql`` should look like this:
+
+::
Review comment:
```suggestion
.. code-block::sql
```
for syntax highlighting
##########
File path: docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst
##########
@@ -0,0 +1,54 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:mssql:
+
+MSSQL Connection
+======================
+The MSSQL connection type enables connection to `MSSQL <https://www.microsoft.com/en-in/sql-server/>`__.
+
+Default Connection IDs
+----------------------
+
+MSSQL Hook uses parameter ``mssql_conn_id`` for Connection IDs and the value of the parameter
+as ``mssql_default`` by default.
Review comment:
```suggestion
MSSQL Hook uses parameter ``mssql_conn_id`` for the Connection ID. The default value
is ``mssql_default``.
```
##########
File path: docs/apache-airflow-providers-microsoft-mssql/connections/mssql.rst
##########
@@ -0,0 +1,54 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+
+
+.. _howto/connection:mssql:
+
+MSSQL Connection
+======================
+The MSSQL connection type enables connection to `MSSQL <https://www.microsoft.com/en-in/sql-server/>`__.
Review comment:
Maybe we should use the “official” name here, i.e. Microsoft SQL Server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Mssql dag
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-963861339
> Cool. Looks good, I would like however to add one more thing (this might be a separate PR but it would be great to add it here).
>
> The comments `[START howto_operator_mssql]` and the like are used to generate HowTo documentation out of the examples. This is cool the example DAG works, but the biggest benefit of it is the documentaiton on using the operators extracts the snippets of that working example dags and puts them in "Howto" guides. For example see how it is done in MySQL: https://github.com/apache/airflow/blob/main/docs/apache-airflow-providers-mysql/operators.rst
>
> It would be great to heave such HowTo document added. Can you add it please too ?
I will add it in this PR as it will make more sense. thank you @potiuk for your feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964480919
Yeah @Bowrna . The Doc errors are a but misleading and require a bit of deeper understanding :
You need to make sure that link to "example_dags" is added to the index - otherwise no-one will be able to find it :):
```
/docs/apache-airflow-providers-microsoft-mssql/index.rst
```
You can see `index.rst` in other providers that alredy have `example_dags` and do it similarly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #19470: Mssql dag
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964088332
> @potiuk I have added the docs and i tried to commit the doc into repository and the pre-commit build fails
>
> ```
> - hook id: provider-yamls
> - exit code: 1
>
> Checking integration duplicates
> Checking completeness of list of {sensors, hooks, operators}
> -- {sensors, hooks, operators} - Expected modules(Left): Current Modules(Right)
> Checking for duplicates in list of {sensors, hooks, operators}
> Checking completeness of list of transfers
> -- Expected transfers modules(Left): Current transfers Modules(Right)
> Checking for duplicates in list of transfers
> Checking connection classes belong to package
> -- Checking providers: present in code(left), mentioned in /Users/sathishkannan/code/airflow/.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml (right)
> Checking doc files
> -- Checking document urls: expected(left), current(right)
> -- Items in the left set but not the right:
> '/docs/apache-airflow-providers-microsoft-mssql/operators.rst'
> ```
>
> I could not understand this failure message. Can you help me to understand this?
Note that it fails on `hook id: provider-yamls` so this gives you context into where the problem is. Something that is related to the provider yaml. Possibly you also edited a howto guide (`.rst` file) but you did not add a corresponded entry in the provider `.yaml`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-968263917
> @Bowrna doc build is failing. can you take a look?
sure i will check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964480919
Yeah @Bowrna . The Doc errors are a but misleading and require a bit of deeper understanding :
You need to make sure that link to "example_dags" is added to the index - otherwise no-one will be able to find it :). It's this file:
```
/docs/apache-airflow-providers-microsoft-mssql/index.rst
```
You can see `index.rst` in other providers that alredy have `example_dags` and do it similarly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964330746
> > @potiuk I have added the docs and i tried to commit the doc into repository and the pre-commit build fails
> > ```
> > - hook id: provider-yamls
> > - exit code: 1
> >
> > Checking integration duplicates
> > Checking completeness of list of {sensors, hooks, operators}
> > -- {sensors, hooks, operators} - Expected modules(Left): Current Modules(Right)
> > Checking for duplicates in list of {sensors, hooks, operators}
> > Checking completeness of list of transfers
> > -- Expected transfers modules(Left): Current transfers Modules(Right)
> > Checking for duplicates in list of transfers
> > Checking connection classes belong to package
> > -- Checking providers: present in code(left), mentioned in /Users/sathishkannan/code/airflow/.github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml (right)
> > Checking doc files
> > -- Checking document urls: expected(left), current(right)
> > -- Items in the left set but not the right:
> > '/docs/apache-airflow-providers-microsoft-mssql/operators.rst'
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > I could not understand this failure message. Can you help me to understand this?
>
> Note that it fails on `hook id: provider-yamls` so this gives you context into where the problem is. Something that is related to the provider yaml. Possibly you also edited a howto guide (`.rst` file) but you did not add a corresponded entry in the provider `.yaml`
@eladkal thank you. I could understand the context of the problem in provider yaml. I have edited the `.rst` file. But I am not sure what I have to add in the provider `.yaml` file. Do I have to increase the version or make any changes to `.yaml` file for adding `.rst` file? If there is any help/guide about this, it would be useful for me to get started.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-968261251
@Bowrna doc build is failing. can you take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #19470:
URL: https://github.com/apache/airflow/pull/19470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
Bowrna commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-968256862
I am not sure why there is a failure in CI checks. Do I have to fix something to get the check pass?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #19470: Add how-to Guide for MSSQL operators
Posted by GitBox <gi...@apache.org>.
eladkal commented on pull request #19470:
URL: https://github.com/apache/airflow/pull/19470#issuecomment-964418802
I can't say for sure because I can't see your code but you can check other providers for example Amazon:
https://github.com/apache/airflow/blob/main/airflow/providers/amazon/provider.yaml#L43
You need to set similar `how-to-guide:` entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org