You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/08 17:35:25 UTC
[GitHub] [airflow] sshah90 opened a new pull request #12925: [Loki log handler] - Integration with Grafana Loki
sshah90 opened a new pull request #12925:
URL: https://github.com/apache/airflow/pull/12925
This Pull request will make the following changes:
Create a log handler that pushes task logs to Loki with labels and using the same labels pull logs to Airflow Web UI.
related: #12920
**Note**:
I had added documentation at [logging-tasks.rst](https://github.com/apache/airflow/blob/master/docs/apache-airflow/logging-monitoring/logging-tasks.rst) file but looks like all log handler docs have been moved to `providers` folders so I make change accordingly and update the same PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] whatnick commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
whatnick commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-789429375
+1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sshah90 commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
sshah90 commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-823460216
I will fix conflicts asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-858830114
Oh. try_number is going to confuse things too;
The the start/end dates in the TI will be for the latest try only, so you'll need to look elsewhere (TaskFail table I think) for those dates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] xinbinhuang commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
xinbinhuang commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-819731970
@sshah90 Welcome to Airflow:) This will be a great feature! Can you rebase and fix the tests please. Let me know if you need helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-859702594
@sshah90 Something like that could work -- though you don't need duplication between the hash and the labels -- each "key" only needs to appear in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#discussion_r648694429
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
+ :type base_log_folder: str
+ :param filename_template: template filename string (incase Loki is down)
+ :type filename_template: str
+ :param loki_conn_id: Connection ID that will be used for authorization to the Loki Platform.
+ :type loki_conn_id: str
+ :param name: the name of the custom log in Loki Logging. Defaults to 'airflow'.
+ :type name: str
+ :param labels: (Optional) Mapping of labels for the entry.
+ :type labels: dict
+ """
+
+ # pylint: disable=too-many-arguments
+ def __init__(
+ self,
+ base_log_folder: str,
+ filename_template: str,
+ loki_conn_id: str,
+ name: str = DEFAULT_LOGGER_NAME,
+ labels: Optional[Dict[str, str]] = None,
+ ):
+ super().__init__(base_log_folder, filename_template)
+ self.loki_conn_id = loki_conn_id
+ self.name: str = name
+ self.timestamp_pattern = "%Y-%m-%dT%H:%M:%S"
+ self.labels = labels
+ self._session: Optional[requests.Session] = None
+
+ @cached_property
+ def get_conn(self):
Review comment:
Since this is a cached property, it should be named like this
```suggestion
def conn(self):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-791561549
[The Workflow run](https://github.com/apache/airflow/actions/runs/625147890) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-745436264
[The Workflow run](https://github.com/apache/airflow/actions/runs/423752262) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-741893837
[The Workflow run](https://github.com/apache/airflow/actions/runs/411002355) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-791587212
[The Workflow run](https://github.com/apache/airflow/actions/runs/625193715) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-858833643
For instance I wonder if you can use something like the JSON output format that ElasticSearch uses, and then combine it with https://grafana.com/docs/loki/latest/logql/#parser-expression to use date-range to limit down the logs returned, and then a pipeline like that to make sure we only get the right logs back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#discussion_r648693851
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
Review comment:
```suggestion
:param base_log_folder: Base log folder to place logs (incase Loki is down).
```
Need a blank line between prose and parameters for it to render correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] weldpua2008 commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
weldpua2008 commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-753965968
Hey @sshah90,
It will be a great feature.
In our case, we are using vector to Loki to gather logs (that has limitations [Issue#3567](https://github.com/timberio/vector/issues/3567).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-746759083
[The Workflow run](https://github.com/apache/airflow/actions/runs/426315159) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-746967259
[The Workflow run](https://github.com/apache/airflow/actions/runs/426533716) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-740786757
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-740805498
[The Workflow run](https://github.com/apache/airflow/actions/runs/408849432) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sshah90 commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
sshah90 commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-858826335
Hi @ashb,
> Not including execution_date in the task labels is incorrect -- the execution date is needed to uniquely identify the task instance
I agree, in some instances, you can't uniquely identify the task. However, adding this label (execution_date) would exponentially increase the cardinality in Loki to the point that it would be inoperable.
This was discussed at great length with the Loki team and therefore, it was intentionally left out. This is also documented [here](https://grafana.com/docs/loki/latest/best-practices/#use-dynamic-labels-sparingly).
> by not including this it means that logs from the task on two separate days would get combined.
This is true however we use `start_time` to filter the log data so that it's not combined when viewing the task logs in the UI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-741002173
[The Workflow run](https://github.com/apache/airflow/actions/runs/409039334) is cancelling this PR. Building image for the PR has been cancelled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-791586809
[The Workflow run](https://github.com/apache/airflow/actions/runs/625255961) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-741001139
[The Workflow run](https://github.com/apache/airflow/actions/runs/409130207) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-746968455
[The Workflow run](https://github.com/apache/airflow/actions/runs/426550753) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-858828896
You can have multiple tasks for different dag runs running concurrently, so filtering for logs between start and end date would not be a complete solution -- you would still get logs from TIs interleaved in one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-823536721
[The Workflow run](https://github.com/apache/airflow/actions/runs/768184459) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sshah90 commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
sshah90 commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-858928062
@ashb
wondering if I can add hash key of `dag_name + task_name + execution_date + try_number` in each log stream with ` setFormatter(self.formatter)` function and filter on that while reading?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nlwstein commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
nlwstein commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-1082260868
Bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on a change in pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
RosterIn commented on a change in pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#discussion_r577741770
##########
File path: docs/apache-airflow-providers-grafana/logging.rst
##########
@@ -0,0 +1,50 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _write-logs-loki:
+
+Writing Logs to Grafana Loki
+----------------------------------
+
+Airflow can be configured to read and write task logs in `Grafana Loki <https://github.com/grafana/loki/>`__.
+It uses an existing Airflow connection to read or write logs. If you don't have a connection properly setup,
+this process will fail.
+
+To enable this feature, ``airflow.cfg`` must be configured as in this
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS S3, Google Cloud Storage, Elastic Search or Loki.
+ # Users must supply an Airflow connection id that provides access to the storage
+ # location. If remote_logging is set to true, see UPDATING.md for additional
+ # configuration requirements.
+ remote_logging = True
+ remote_base_log_folder = 'loki:///airflow'
+ remote_log_conn_id = <name of the Loki connection>
+
+All configuration options are in the ``[logging]`` section.
+
+The value of field ``remote_logging`` must always be set to ``True`` for this feature to work.
+Turning this option off will result in data not being sent to Loki.
Review comment:
you need to add loki to spelling list as well
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-898053846
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #12925:
URL: https://github.com/apache/airflow/pull/12925
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] whatnick commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
whatnick commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-823173073
I would love this feature our workers in Kubernetes sometimes die and we lose logs. Streaming logs via Loki would be great. Please let me know how I can help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-741893296
[The Workflow run](https://github.com/apache/airflow/actions/runs/411069392) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-758882669
[The Workflow run](https://github.com/apache/airflow/actions/runs/480881483) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] whatnick edited a comment on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
whatnick edited a comment on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-789429375
+1 I would be interested in contributing and progressing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sshah90 commented on a change in pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
sshah90 commented on a change in pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#discussion_r588455010
##########
File path: docs/apache-airflow-providers-grafana/logging.rst
##########
@@ -0,0 +1,50 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+.. _write-logs-loki:
+
+Writing Logs to Grafana Loki
+----------------------------------
+
+Airflow can be configured to read and write task logs in `Grafana Loki <https://github.com/grafana/loki/>`__.
+It uses an existing Airflow connection to read or write logs. If you don't have a connection properly setup,
+this process will fail.
+
+To enable this feature, ``airflow.cfg`` must be configured as in this
+
+.. code-block:: ini
+
+ [logging]
+ # Airflow can store logs remotely in AWS S3, Google Cloud Storage, Elastic Search or Loki.
+ # Users must supply an Airflow connection id that provides access to the storage
+ # location. If remote_logging is set to true, see UPDATING.md for additional
+ # configuration requirements.
+ remote_logging = True
+ remote_base_log_folder = 'loki:///airflow'
+ remote_log_conn_id = <name of the Loki connection>
+
+All configuration options are in the ``[logging]`` section.
+
+The value of field ``remote_logging`` must always be set to ``True`` for this feature to work.
+Turning this option off will result in data not being sent to Loki.
Review comment:
Hi @RosterIn
In my local static check never failed but I added it just in case.
Thanks for the suggestion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#discussion_r648695501
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
+ :type base_log_folder: str
+ :param filename_template: template filename string (incase Loki is down)
+ :type filename_template: str
+ :param loki_conn_id: Connection ID that will be used for authorization to the Loki Platform.
+ :type loki_conn_id: str
+ :param name: the name of the custom log in Loki Logging. Defaults to 'airflow'.
+ :type name: str
+ :param labels: (Optional) Mapping of labels for the entry.
+ :type labels: dict
+ """
+
+ # pylint: disable=too-many-arguments
+ def __init__(
+ self,
+ base_log_folder: str,
+ filename_template: str,
+ loki_conn_id: str,
+ name: str = DEFAULT_LOGGER_NAME,
+ labels: Optional[Dict[str, str]] = None,
+ ):
+ super().__init__(base_log_folder, filename_template)
+ self.loki_conn_id = loki_conn_id
+ self.name: str = name
+ self.timestamp_pattern = "%Y-%m-%dT%H:%M:%S"
+ self.labels = labels
+ self._session: Optional[requests.Session] = None
+
+ @cached_property
+ def get_conn(self):
+ """Loki connection for client"""
+ return BaseHook.get_connection(self.loki_conn_id)
+
+ @property
+ def session(self) -> requests.Session:
+ """Create HTTP session"""
+ if self._session is None:
+ self._session = requests.Session()
+ self._session.auth = (self.get_conn.login, self.get_conn.password) or None
+ return self._session
+
+ def is_loki_alive(self):
+ """Checks whether Loki is ready for pushing/pulling logs"""
+ try:
+ status = self.session.get(
+ f"{self.get_conn.host}/ready",
+ )
Review comment:
```suggestion
status = self.http_session.get(f"{self.get_conn.host}/ready")
```
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
+ :type base_log_folder: str
+ :param filename_template: template filename string (incase Loki is down)
+ :type filename_template: str
+ :param loki_conn_id: Connection ID that will be used for authorization to the Loki Platform.
+ :type loki_conn_id: str
+ :param name: the name of the custom log in Loki Logging. Defaults to 'airflow'.
+ :type name: str
+ :param labels: (Optional) Mapping of labels for the entry.
+ :type labels: dict
+ """
+
+ # pylint: disable=too-many-arguments
+ def __init__(
+ self,
+ base_log_folder: str,
+ filename_template: str,
+ loki_conn_id: str,
+ name: str = DEFAULT_LOGGER_NAME,
+ labels: Optional[Dict[str, str]] = None,
+ ):
+ super().__init__(base_log_folder, filename_template)
+ self.loki_conn_id = loki_conn_id
+ self.name: str = name
+ self.timestamp_pattern = "%Y-%m-%dT%H:%M:%S"
+ self.labels = labels
+ self._session: Optional[requests.Session] = None
+
+ @cached_property
+ def get_conn(self):
+ """Loki connection for client"""
+ return BaseHook.get_connection(self.loki_conn_id)
+
+ @property
+ def session(self) -> requests.Session:
+ """Create HTTP session"""
+ if self._session is None:
+ self._session = requests.Session()
+ self._session.auth = (self.get_conn.login, self.get_conn.password) or None
+ return self._session
Review comment:
Session in Airflow commonly refers to the SQLAlchemy DB session, so lets rename this:
```suggestion
@cached_property
def http_session(self) -> requests.Session:
"""Create HTTP session"""
session = requests.Session()
session.auth = (self.get_conn.login, self.get_conn.password) or None
return session
```
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
+ :type base_log_folder: str
+ :param filename_template: template filename string (incase Loki is down)
+ :type filename_template: str
+ :param loki_conn_id: Connection ID that will be used for authorization to the Loki Platform.
+ :type loki_conn_id: str
+ :param name: the name of the custom log in Loki Logging. Defaults to 'airflow'.
+ :type name: str
+ :param labels: (Optional) Mapping of labels for the entry.
+ :type labels: dict
+ """
+
+ # pylint: disable=too-many-arguments
+ def __init__(
+ self,
+ base_log_folder: str,
+ filename_template: str,
+ loki_conn_id: str,
+ name: str = DEFAULT_LOGGER_NAME,
+ labels: Optional[Dict[str, str]] = None,
+ ):
+ super().__init__(base_log_folder, filename_template)
+ self.loki_conn_id = loki_conn_id
+ self.name: str = name
+ self.timestamp_pattern = "%Y-%m-%dT%H:%M:%S"
+ self.labels = labels
+ self._session: Optional[requests.Session] = None
+
+ @cached_property
+ def get_conn(self):
+ """Loki connection for client"""
+ return BaseHook.get_connection(self.loki_conn_id)
+
+ @property
+ def session(self) -> requests.Session:
+ """Create HTTP session"""
+ if self._session is None:
+ self._session = requests.Session()
Review comment:
You should set a timeout on this session, otherwise the `loki_is_alive()` check may hang for a long long time.
##########
File path: airflow/providers/grafana/log/loki_task_handler.py
##########
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+"""Loki logging handler for tasks"""
+import time
+from typing import Dict, Optional, Tuple, Union
+
+import logging_loki
+import requests
+from cached_property import cached_property
+
+from airflow.hooks.base_hook import BaseHook
+from airflow.models import TaskInstance
+from airflow.utils.log.file_task_handler import FileTaskHandler
+from airflow.utils.log.logging_mixin import LoggingMixin
+
+DEFAULT_LOGGER_NAME = "airflow"
+
+
+class LokiTaskHandler(FileTaskHandler, LoggingMixin):
+ """
+ LokiTaskHandler that directly makes Loki logging API calls while reading and writing logs.
+ This is a Python standard ``logging`` handler using that can be used to route Python standard
+ logging messages directly to the Loki Logging API. It can also be used to save logs for
+ executing tasks. To do this, you should set as a handler with the name "tasks". In this case,
+ it will also be used to read the log for display in Web UI.
+ :param base_log_folder: Base log folder to place logs (incase Loki is down).
+ :type base_log_folder: str
+ :param filename_template: template filename string (incase Loki is down)
+ :type filename_template: str
+ :param loki_conn_id: Connection ID that will be used for authorization to the Loki Platform.
+ :type loki_conn_id: str
+ :param name: the name of the custom log in Loki Logging. Defaults to 'airflow'.
+ :type name: str
+ :param labels: (Optional) Mapping of labels for the entry.
+ :type labels: dict
+ """
+
+ # pylint: disable=too-many-arguments
+ def __init__(
+ self,
+ base_log_folder: str,
+ filename_template: str,
+ loki_conn_id: str,
+ name: str = DEFAULT_LOGGER_NAME,
+ labels: Optional[Dict[str, str]] = None,
+ ):
+ super().__init__(base_log_folder, filename_template)
+ self.loki_conn_id = loki_conn_id
+ self.name: str = name
+ self.timestamp_pattern = "%Y-%m-%dT%H:%M:%S"
+ self.labels = labels
+ self._session: Optional[requests.Session] = None
+
+ @cached_property
+ def get_conn(self):
+ """Loki connection for client"""
+ return BaseHook.get_connection(self.loki_conn_id)
+
+ @property
+ def session(self) -> requests.Session:
+ """Create HTTP session"""
+ if self._session is None:
+ self._session = requests.Session()
+ self._session.auth = (self.get_conn.login, self.get_conn.password) or None
+ return self._session
+
+ def is_loki_alive(self):
+ """Checks whether Loki is ready for pushing/pulling logs"""
+ try:
+ status = self.session.get(
+ f"{self.get_conn.host}/ready",
+ )
+ return status.status_code
+ except ConnectionError as error_msg:
+ self.log.exception(error_msg)
+ return None
+
+ @staticmethod
+ def _task_label(task_instance: TaskInstance) -> Dict[str, str]:
+ """
+ Returns task instance labels for Loki which will use while reading
+ and writing logs from loki.
+ :param task_instance: task instance object
+ :type: task_instance: TaskInstance
+ """
+ # Not adding execution date since it violates Loki label standards
+ # https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/
+
+ return {
+ "airflow_dag_id": task_instance.dag_id,
+ "airflow_task_id": task_instance.task_id,
+ "airflow_try_number": str(task_instance.try_number),
+ }
+
+ def get_label(self, task_instance: TaskInstance) -> Dict[str, str]:
+ """
+ Update task_labels with optional labels and return Loki labels.
+ :param task_instance: task instance object
+ :type: task_instance: TaskInstance
+ """
+ tags = {}
+ task_labels = self._task_label(task_instance)
+ if self.labels:
+ tags.update(self.labels)
+ tags.update(task_labels)
+ return tags
+ return task_labels
Review comment:
```suggestion
if self.labels:
tags = copy.copy(self.labels)
else:
tags = {}
tags.update(self._task_label(task_instance))
return tags
```
You'll need to `import copy` too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] whatnick commented on pull request #12925: [Loki log handler] - Integration with Grafana Loki
Posted by GitBox <gi...@apache.org>.
whatnick commented on pull request #12925:
URL: https://github.com/apache/airflow/pull/12925#issuecomment-903173747
Bump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org