You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/12 15:36:05 UTC
[GitHub] [airflow] francescomucio opened a new pull request #11472: Jdbc Operator How To
francescomucio opened a new pull request #11472:
URL: https://github.com/apache/airflow/pull/11472
Added an How To page for the Jdbc operator (and an example dag to get code snippet for the page)
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-707283903
[The Workflow run](https://github.com/apache/airflow/actions/runs/302738747) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r509076857
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
+
+ * Install a JVM.
+ * Have the JDBC driver for your database installed.
Review comment:
I have included a simple check, not sure if it is enough
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504524081
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
Review comment:
In the software section like for ODBC?
Should I add this as part of this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-719026828
Sorry for the mess, not sure what I should keep an eye on next time
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-718888659
@potiuk Selective tests (? or something else) caused CI to not detect failing tests:
https://github.com/apache/airflow/runs/1307780299
```
tests/core/test_project_structure.py sssssssssss [ 23%]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r510459185
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,97 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Java Database Connectivity (JDBC) is an application programming interface
+(API) for the programming language Java, which defines how a client may
+access a database.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
Review comment:
```suggestion
pip install 'apache-airflow[jdbc]'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj merged pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #11472:
URL: https://github.com/apache/airflow/pull/11472
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-707284245
[The Workflow run](https://github.com/apache/airflow/actions/runs/302744846) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504322692
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
+
+ * Install a JVM.
+ * Have the JDBC driver for your database installed.
Review comment:
Can you add a link to external docs about it? Ideally, you should include a simple command here to check if this condition is met.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r509076360
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
Review comment:
moved
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504321773
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
Review comment:
This should be the second section in the document and the first should describe the JDBC.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504652335
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
+
+ * Install a JVM.
Review comment:
I always found installing Java very platform specific, what do you think of https://adoptopenjdk.net/installation.html?
I guess it is missing the step about setting the JAVA_HOME folder like you do in [your Dockerfile](https://gist.github.com/mik-laj/c090f51c970e4f6898e6fd986ea1e12e#file-java-hadoop-dockerfile-L18).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r511795272
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -75,6 +75,8 @@ Fundamentals
* - :mod:`airflow.operators.latest_only`
-
+ * - :mod:`airflow.providers.jdbc.operators.jdbc`
Review comment:
Thanks for pointing out this, I removed this and moved the link to the JDBC doc below
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] francescomucio commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
francescomucio commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r511794766
##########
File path: airflow/example_dags/example_jdbc_operator.py
##########
@@ -0,0 +1,66 @@
+#
Review comment:
moved
##########
File path: airflow/example_dags/example_jdbc_operator.py
##########
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Example DAG demonstrating the usage of the BashOperator."""
+
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.providers.jdbc.operators.jdbc import JdbcOperator
+from airflow.utils.dates import days_ago
+
+args = {
+ 'owner': 'airflow',
+}
+
+with DAG(
+ dag_id='example_jdbc_operator',
+ default_args=args,
+ schedule_interval='0 0 * * *',
+ start_date=days_ago(2),
+ dagrun_timeout=timedelta(minutes=60),
+ tags=['example']
+) as dag:
+
+ run_this_last = DummyOperator(
+ task_id='run_this_last',
+ dag=dag,
+ )
+
+ # [START howto_operator_jdbc_template]
+ delete_data = JdbcOperator(
+ task_id='insert_data',
Review comment:
thanks, fixed
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,97 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Java Database Connectivity (JDBC) is an application programming interface
+(API) for the programming language Java, which defines how a client may
+access a database.
+
+.. contents::
+ :depth: 1
+ :local:
+
+Prerequisite Tasks
+^^^^^^^^^^^^^^^^^^
+
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
Review comment:
modified
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-718887707
This, I think, has broken master and all open PRs
```
> self.assertEqual([], invalid_files)
E AssertionError: Lists differ: [] != ['/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py']
E
E Second list contains 1 additional elements.
E First extra element 0:
E '/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py'
E
E - []
E + ['/opt/airflow/airflow/providers/jdbc/example_dags/example_jdbc_operator.py']
tests/core/test_project_structure.py:230: AssertionError
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-707428724
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-707284057
[The Workflow run](https://github.com/apache/airflow/actions/runs/302731005) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r510218172
##########
File path: airflow/example_dags/example_jdbc_operator.py
##########
@@ -0,0 +1,66 @@
+#
Review comment:
This file should be in `airflow.providers.jdbc.example_dags`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504322341
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
+
+ * Install a JVM.
Review comment:
Can you add a link to external docs about it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-718847434
Good work @francescomucio 🎉
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r504321387
##########
File path: docs/howto/operator/jdbc.rst
##########
@@ -0,0 +1,62 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _howto/operator:JdbcOperator:
+
+JdbcOperator
+============
+
+Use the :class:`~airflow.providers.jdbc.operators.jdbc` to execute
+commands against a database (or data storage) accessible via a JDBC driver.
+
+The :doc:`JDBC Connection </howto/connection/jdbc>` must be passed as
+``jdbc_conn_id``.
+
+.. exampleinclude:: /../airflow/example_dags/example_jdbc_operator.py
+ :language: python
+ :start-after: [START howto_operator_jdbc]
+ :end-before: [END howto_operator_jdbc]
+
+The parameter ``sql`` can receive a string or a list of strings.
+Each string can be an SQL statement or a reference to a template file.
+Template reference are recognized by ending in '.sql'.
+
+The parameter ``autocommit`` if set to ``True`` will execute a commit after
+each command (default is ``False``)
+
+Prerequisite Tasks
+------------------
+To use this operator you need:
+
+ * Install the python module jaydebeapi:
+ .. code-block:: bash
+
+ pip install jaydebeapi
Review comment:
I think, we should use extra packages to manage dependencies for Airflow. WDYT?
See: https://airflow.readthedocs.io/en/latest/installation.html#extra-packages
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r510458672
##########
File path: airflow/example_dags/example_jdbc_operator.py
##########
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Example DAG demonstrating the usage of the BashOperator."""
+
+from datetime import timedelta
+
+from airflow import DAG
+from airflow.operators.dummy_operator import DummyOperator
+from airflow.providers.jdbc.operators.jdbc import JdbcOperator
+from airflow.utils.dates import days_ago
+
+args = {
+ 'owner': 'airflow',
+}
+
+with DAG(
+ dag_id='example_jdbc_operator',
+ default_args=args,
+ schedule_interval='0 0 * * *',
+ start_date=days_ago(2),
+ dagrun_timeout=timedelta(minutes=60),
+ tags=['example']
+) as dag:
+
+ run_this_last = DummyOperator(
+ task_id='run_this_last',
+ dag=dag,
+ )
+
+ # [START howto_operator_jdbc_template]
+ delete_data = JdbcOperator(
+ task_id='insert_data',
Review comment:
```suggestion
task_id='delete_data',
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#discussion_r510460229
##########
File path: docs/operators-and-hooks-ref.rst
##########
@@ -75,6 +75,8 @@ Fundamentals
* - :mod:`airflow.operators.latest_only`
-
+ * - :mod:`airflow.providers.jdbc.operators.jdbc`
Review comment:
The fundamentals section only includes operators that are in the Airflow core. Please look at the protocol section: https://airflow.readthedocs.io/en/latest/operators-and-hooks-ref.html#protocol-operators-and-hooks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #11472: Jdbc Operator How To
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11472:
URL: https://github.com/apache/airflow/pull/11472#issuecomment-707428864
[The Workflow run](https://github.com/apache/airflow/actions/runs/303438842) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks$,^Build docs$,^Spell check docs$,^Backport packages$,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org