You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/01 17:56:34 UTC
[GitHub] [airflow] dstandish opened a new pull request #21255: Clarify ElasticsearchTaskHandler docstring
dstandish opened a new pull request #21255:
URL: https://github.com/apache/airflow/pull/21255
Previously it said 'logs are not indexed into ES' but what it meant was '_airflow_ does not index the logs for you'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #21255: Clarify ElasticsearchTaskHandler docstring
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #21255:
URL: https://github.com/apache/airflow/pull/21255#issuecomment-1027174245
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #21255: Clarify ElasticsearchTaskHandler docstring
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #21255:
URL: https://github.com/apache/airflow/pull/21255#discussion_r796859257
##########
File path: airflow/providers/elasticsearch/log/es_task_handler.py
##########
@@ -44,17 +44,17 @@
class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMixin):
"""
ElasticsearchTaskHandler is a python log handler that
- reads logs from Elasticsearch. Note logs are not directly
- indexed into Elasticsearch. Instead, it flushes logs
+ reads logs from Elasticsearch. Note that Airflow does not handle the indexing
+ of logs into Elasticsearch. Instead, Airflow flushes logs
into local files. Additional software setup is required
- to index the log into Elasticsearch, such as using
+ to index the logs into Elasticsearch, such as using
Filebeat and Logstash.
- To efficiently query and sort Elasticsearch results, we assume each
+ To efficiently query and sort Elasticsearch results, this handler assumes each
log message has a field `log_id` consists of ti primary keys:
`log_id = {dag_id}-{task_id}-{execution_date}-{try_number}`
Log messages with specific log_id are sorted based on `offset`,
Review comment:
```suggestion
Log messages are sorted based on `offset`,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #21255: Clarify ElasticsearchTaskHandler docstring
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #21255:
URL: https://github.com/apache/airflow/pull/21255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #21255: Clarify ElasticsearchTaskHandler docstring
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #21255:
URL: https://github.com/apache/airflow/pull/21255#discussion_r796858720
##########
File path: airflow/providers/elasticsearch/log/es_task_handler.py
##########
@@ -44,17 +44,17 @@
class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMixin):
"""
ElasticsearchTaskHandler is a python log handler that
- reads logs from Elasticsearch. Note logs are not directly
- indexed into Elasticsearch. Instead, it flushes logs
+ reads logs from Elasticsearch. Note that Airflow does not handle the indexing
+ of logs into Elasticsearch. Instead, Airflow flushes logs
into local files. Additional software setup is required
- to index the log into Elasticsearch, such as using
+ to index the logs into Elasticsearch, such as using
Filebeat and Logstash.
- To efficiently query and sort Elasticsearch results, we assume each
+ To efficiently query and sort Elasticsearch results, this handler assumes each
log message has a field `log_id` consists of ti primary keys:
Review comment:
```suggestion
log message has a field `log_id` consisting of ti's primary keys:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #21255: Clarify ElasticsearchTaskHandler docstring
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #21255:
URL: https://github.com/apache/airflow/pull/21255#discussion_r796859486
##########
File path: airflow/providers/elasticsearch/log/es_task_handler.py
##########
@@ -44,17 +44,17 @@
class ElasticsearchTaskHandler(FileTaskHandler, ExternalLoggingMixin, LoggingMixin):
"""
ElasticsearchTaskHandler is a python log handler that
- reads logs from Elasticsearch. Note logs are not directly
- indexed into Elasticsearch. Instead, it flushes logs
+ reads logs from Elasticsearch. Note that Airflow does not handle the indexing
+ of logs into Elasticsearch. Instead, Airflow flushes logs
into local files. Additional software setup is required
- to index the log into Elasticsearch, such as using
+ to index the logs into Elasticsearch, such as using
Filebeat and Logstash.
- To efficiently query and sort Elasticsearch results, we assume each
+ To efficiently query and sort Elasticsearch results, this handler assumes each
log message has a field `log_id` consists of ti primary keys:
`log_id = {dag_id}-{task_id}-{execution_date}-{try_number}`
Log messages with specific log_id are sorted based on `offset`,
which is a unique integer indicates log message's order.
Review comment:
```suggestion
which is a unique integer indicating the log message's order.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org