You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/04/07 11:05:01 UTC
[GitHub] [airflow] Bowrna opened a new pull request, #22811: AIP-47 es new system tests
Bowrna opened a new pull request, #22811:
URL: https://github.com/apache/airflow/pull/22811
closes: #22445
relates: #22445
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
bhirsz commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1098303204
> LGTM. @bhirsz ?
@potiuk LGTM too. Good PR @Bowrna!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #22811:
URL: https://github.com/apache/airflow/pull/22811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on a diff in pull request #22811: AIP-47 es new system tests
Posted by GitBox <gi...@apache.org>.
Bowrna commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r846567161
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
Review Comment:
@ferruzzi I followed the steps mentioned in this doc to design the system.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-47+New+design+of+Airflow+System+Tests#AIP47NewdesignofAirflowSystemTests-design_details
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #22811: AIP-47 es new system tests
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r846478487
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
Review Comment:
Is there a particular reason you moved these globals? They aren't being pulled from ENV and are each only used once, seems perfectly reasonable to me to keep them inlined as they were in the sample DAG?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1095346163
> I am not sure why static check fails here. If anyone knows how to fix this, can you tell me?
Looks like it's a problem with main, not your code. You can make it retry by closing and reopening the PR, or it will try again when you add a new commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] joppevos commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
joppevos commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1094329235
> I get the following error when I run the test using pytest in breeze environment for Elasticsearch. Could you share your insights on where I am going wrong?
First of all, good that you mention this issue!
I got the same error, but only after running the tests for the second time. I think there is some state being written somewhere, but I have not figured out yet where.
My hacky solution is to run the tests within breeze. Then every iteration you restart the container and re-run the tests.
This is far from ideal, so I would also be interested to hear the opinions of others, before I dive in to figure it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ferruzzi commented on a diff in pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
ferruzzi commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r847509399
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
Review Comment:
My mistake. I forgot the AIP suggested moving them to globals. Carry on :stuck_out_tongue:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on a diff in pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
bhirsz commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r846961752
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
+
+
+@task(task_id='es_print_tables')
+def show_tables():
+ """
+ show_tables queries elasticsearch to list available tables
+ """
+ es = ElasticsearchHook(elasticsearch_conn_id=CONN_ID)
+
+ # Handle ES conn with context manager
+ with es.get_conn() as es_conn:
+ tables = es_conn.execute('SHOW TABLES')
+ for table, *_ in tables:
+ print(f"table: {table}")
+ return True
+
+
+with models.DAG(
+ DAG_ID,
+ schedule_interval="@once",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=["example", "elasticsearch"],
+) as dag:
+ # [START howto_elasticsearch_query]
+ execute_query = show_tables()
Review Comment:
Note that sphinx will only take content between START and END markers. So your documentation will show:
```
execute_query = show_tables()
```
which is not telling much. Unless it's what you intendent consider moving the markers to where you define the show_tables:
```
[START howto_elasticsearch_query]
@task(task_id='es_print_tables')
def show_tables():
(method body)
[END howto_elasticsearch_query]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #22811: AIP-47 es new system tests
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1093915480
I am not sure why static check fails here. If anyone knows how to fix this, can you tell me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
bhirsz commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1094572808
>
It was also noticed in my team last week unfortunately - one of the workaround was to restart the Breeze environment like you mention. The other workaround is to use a unique DAG id between runs (something along DAG_ID = f"name_{uuid.uuid4()}". I will look into this issue and see how it can be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1098307522
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on a diff in pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
bhirsz commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r846961752
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
+
+
+@task(task_id='es_print_tables')
+def show_tables():
+ """
+ show_tables queries elasticsearch to list available tables
+ """
+ es = ElasticsearchHook(elasticsearch_conn_id=CONN_ID)
+
+ # Handle ES conn with context manager
+ with es.get_conn() as es_conn:
+ tables = es_conn.execute('SHOW TABLES')
+ for table, *_ in tables:
+ print(f"table: {table}")
+ return True
+
+
+with models.DAG(
+ DAG_ID,
+ schedule_interval="@once",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=["example", "elasticsearch"],
+) as dag:
+ # [START howto_elasticsearch_query]
+ execute_query = show_tables()
Review Comment:
Note that sphinx will only take content between START and END markers. So your documentation will show:
```
execute_query = show_tables()
```
which is not telling much. Consider moving the markers to where you define the show_tables:
```
[START howto_elasticsearch_query]
@task(task_id='es_print_tables')
def show_tables():
(method body)
[END howto_elasticsearch_query]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna commented on pull request #22811: AIP-47 es new system tests
Posted by GitBox <gi...@apache.org>.
Bowrna commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1092007009
I get the following error when I run the test using pytest in breeze environment for Elasticsearch. Could you share your insights on where I am going wrong? I have configured to run the ES in local env and updated the username and password for `elasticsearch_default` conn_id
@potiuk
```
root@20f758b954e9:/opt/airflow/tests/system/providers/elasticsearch# pytest --system elasticsearch example_elasticsearch_query.py
============================= test session starts ==============================
platform linux -- Python 3.7.13, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow, configfile: pytest.ini
plugins: timeouts-1.2.1, httpx-0.20.0, asyncio-0.18.3, instafail-0.4.2, anyio-3.5.0, requests-mock-1.9.3, cov-3.0.0, flaky-3.7.0, rerunfailures-9.1.1, forked-1.4.0, xdist-2.5.0
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collecting ... collected 1 item
example_elasticsearch_query.py::test_run <- tests/system/utils/__init__.py FAILED [100%]
=================================== FAILURES ===================================
___________________________________ test_run ___________________________________
ValueError: None is not a valid DagRunState
During handling of the above exception, another exception occurred:
def test_run():
> dag.clear(dag_run_state=State.NONE)
../../utils/__init__.py:22:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../airflow/utils/session.py:71: in wrapper
return func(*args, session=session, **kwargs)
../../../../airflow/models/dag.py:1856: in clear
dag_run_state=dag_run_state,
../../../../airflow/models/taskinstance.py:289: in clear_task_instances
dag_run_state = DagRunState(dag_run_state) # Validate the state value.
/usr/local/lib/python3.7/enum.py:315: in __call__
return cls.__new__(cls, value)
/usr/local/lib/python3.7/enum.py:569: in __new__
raise exc
/usr/local/lib/python3.7/enum.py:553: in __new__
result = cls._missing_(value)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <enum 'DagRunState'>, value = None
@classmethod
def _missing_(cls, value):
> raise ValueError("%r is not a valid %s" % (value, cls.__name__))
E ValueError: None is not a valid DagRunState
/usr/local/lib/python3.7/enum.py:582: ValueError
---------------------------- Captured stdout setup -----------------------------
========================= AIRFLOW ==========================
Home of the user: /root
Airflow home /root/airflow
Skipping initializing of the DB as it was initialized already.
You can re-initialize the database by adding --with-db-init flag when running tests.
=============================== warnings summary ===============================
../../../../airflow/configuration.py:398
/opt/airflow/airflow/configuration.py:398: FutureWarning: The 'dag_default_view' setting in [webserver] has the old default value of 'tree'. This value has been changed to 'grid' in the running config, but please update your config before Apache Airflow 3.0.
FutureWarning,
../../../../airflow/configuration.py:398
/opt/airflow/airflow/configuration.py:398: FutureWarning: The 'log_filename_template' setting in [logging] has the old default value of '{{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log'. This value has been changed to 'dag_id={{ ti.dag_id }}/run_id={{ ti.run_id }}/task_id={{ ti.task_id }}/{%% if ti.map_index >= 0 %%}map_index={{ ti.map_index }}/{%% endif %%}attempt={{ try_number }}.log' in the running config, but please update your config before Apache Airflow 3.0.
FutureWarning,
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED example_elasticsearch_query.py::test_run - ValueError: None is not a v...
======================== 1 failed, 2 warnings in 1.15s =========================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #22811:
URL: https://github.com/apache/airflow/pull/22811#discussion_r849701921
##########
tests/system/providers/elasticsearch/example_elasticsearch_query.py:
##########
@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""
+Example Airflow DAG for Elasticsearch Query.
+"""
+import os
+from datetime import datetime
+
+from airflow import models
+from airflow.decorators import task
+from airflow.providers.elasticsearch.hooks.elasticsearch import ElasticsearchHook
+
+ENV_ID = os.environ.get("SYSTEM_TESTS_ENV_ID")
+DAG_ID = 'elasticsearch_dag'
+CONN_ID = 'elasticsearch_default'
+
+
+@task(task_id='es_print_tables')
+def show_tables():
+ """
+ show_tables queries elasticsearch to list available tables
+ """
+ es = ElasticsearchHook(elasticsearch_conn_id=CONN_ID)
+
+ # Handle ES conn with context manager
+ with es.get_conn() as es_conn:
+ tables = es_conn.execute('SHOW TABLES')
+ for table, *_ in tables:
+ print(f"table: {table}")
+ return True
+
+
+with models.DAG(
+ DAG_ID,
+ schedule_interval="@once",
+ start_date=datetime(2021, 1, 1),
+ catchup=False,
+ tags=["example", "elasticsearch"],
+) as dag:
+ # [START howto_elasticsearch_query]
+ execute_query = show_tables()
Review Comment:
Yep. Moving them to inside of the method works even better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Bowrna closed pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
Bowrna closed pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
URL: https://github.com/apache/airflow/pull/22811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bhirsz commented on pull request #22811: AIP-47 Migrate ElasticSearch into new system tests
Posted by GitBox <gi...@apache.org>.
bhirsz commented on PR #22811:
URL: https://github.com/apache/airflow/pull/22811#issuecomment-1095089936
@Bowrna @joppevos We used State.None to clear the state of the DAG but it's restricted only to tasks:
https://github.com/apache/airflow/blob/a3dd8473e4c5bbea214ebc8d5545b75281166428/airflow/utils/state.py#L90
I think default state for the DAG should be used instead -> DagRunState.QUEUED. I will update my PR with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org