You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ivanp221 (via GitHub)" <gi...@apache.org> on 2024/04/30 10:29:39 UTC
[I] Logs from elastic not visible in Grid menu [airflow]
ivanp221 opened a new issue, #39323:
URL: https://github.com/apache/airflow/issues/39323
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.6.2
### What happened?
When opening task log from grid menu - loading animation is played and logs won't show. Webserver logs show:
`webserver [2024-04-30T09:51:42.825+0000] {base.py:274} DEBUG - < {"took":8,"timed_out":false,"_shards":{"total":53,"successful":53,"skipped":51,"failed":0},"hits":{"total":{"value":17,"relation":"eq"},"max_score":null,"hits":[{"_index": ....... `
Here I can see all the actual logs that are in elastic an then webserver just bombards elastic with:
`webserver [2024-04-30T09:48:34.688+0000] {base.py:270} INFO - POST https://elastic:9200/_all/_count [status:200 request:0.009s]
webserver [2024-04-30T09:48:34.688+0000] {base.py:273} DEBUG - > {"query":{"bool":{"filter":[{"range":{"offset":{"gt":1714470190001012992}}}],"must":[{"match_phrase":{"log_id":"Redis_Populate_Cities_populate_redis_cities_2024_04_30T09_43_06_661556_1"}}]}}}
webserver [2024-04-30T09:48:34.689+0000] {base.py:274} DEBUG - < {"count":0,"_shards":{"total":53,"successful":53,"skipped":0,"failed":0}}`
Here is my config from values.yaml regarding elastic:
```
elasticsearch:
enabled: true
secretName: airflow-elastic-secret
config:
elasticsearch:
end_of_log_mark: end_of_log
json_fields: asctime, filename, lineno, levelname, message
json_format: "True"
log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
write_stdout: "True"
elasticsearch_configs:
max_retries: 2
retry_timeout: "True"
timeout: 10
use_ssl: "True"
verify_certs: "False"
```
### What you think should happen instead?
_No response_
### How to reproduce
Can be reproduced by clicking dag name in ui and then from grid selecting task and selecting task's logs:
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow==2.6.2
apache-airflow-providers-amazon==8.1.0
apache-airflow-providers-celery==3.2.0
apache-airflow-providers-cncf-kubernetes==7.0.0
apache-airflow-providers-common-sql==1.5.1
apache-airflow-providers-docker==3.7.0
apache-airflow-providers-elasticsearch==4.5.0
apache-airflow-providers-ftp==3.4.1
apache-airflow-providers-google==10.1.1
apache-airflow-providers-grpc==3.2.0
apache-airflow-providers-hashicorp==3.4.0
apache-airflow-providers-http==4.4.1
apache-airflow-providers-imap==3.2.1
apache-airflow-providers-microsoft-azure==6.1.1
apache-airflow-providers-microsoft-mssql==3.4.1
apache-airflow-providers-mongo==3.2.1
apache-airflow-providers-mysql==5.1.0
apache-airflow-providers-odbc==3.3.0
apache-airflow-providers-postgres==5.5.0
apache-airflow-providers-redis==3.2.0
apache-airflow-providers-sendgrid==3.2.0
apache-airflow-providers-sftp==4.3.1
apache-airflow-providers-slack==7.3.0
apache-airflow-providers-snowflake==4.1.0
apache-airflow-providers-sqlite==3.4.1
apache-airflow-providers-ssh==3.7.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Airflow is deployed with official helm chart with changes in values.yaml Dockerfile is customized with pip installs
Please let me know if any additional info is required
### Anything else?
Couple of caveats:
1) When selecting "See More" log is shown.
2) Log also can be viewed from Graph - Task - Log. (But here i have another bug - my logs doesnt contain end_of_log so it keeps polling elastic.)
3) When dag is in running state - i can see its log in grid view.
4) It's only pending when there is a log in elastic (and i can see it with right log_id ) if there is no log in elastic - message will show Log not found in Elasticsearch... etc
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Mant1ss (via GitHub)" <gi...@apache.org>.
Mant1ss commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2296840738
airflow 2.9.0
apache-airflow-providers-elasticsearch 5.3.3
same problem.
There is no "end_of_log" in elastic.
/log? page show log and infinite reload (can't get end_of_log)
dags/{dag_id}/grid? show failed status and webserver OOM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298171683
Also no problem. I wrote a simple DAG that executes a `SHOW TABLE` query :
```
def query_elasticsearch():
es = ElasticsearchSQLHook(elasticsearch_conn_id='elasticsearch_default')
es_connection = es.get_conn()
response = es_connection.execute_sql("SHOW TABLES")
for i in range(5000):
for row in response["rows"]:
print(f"row: {row}")
```
This results in a log file which consists of around 35k log lines:
```
airflow git:(main) ✗ cat logs/dag_id=elasticsearch_sql_dag/run_id=manual__2024-08-20T07:19:51.268943+00:00/task_id=run_es_query/attempt=1.log | wc -l
35028
```
And the UI is still able to render it :
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2850639129
Picking this up.
Sometimes, it is possible that the end_of_log marker can be missing due to an issue with export. When this happens, I think, we should not just keep looping until we find the `end_of_log` marker. We should timeout and probably show a warning that "the logs we show could be incomplete as `end_of_log` marker was not found"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "ivanp221 (via GitHub)" <gi...@apache.org>.
ivanp221 commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2093266308
Sorry for the delay - was having trouble with updating to 2.9.0.
Tested it with latest provider on 2.6.2 apache-airflow-providers-elasticsearch==5.3.4 - same
Tested with airflow 2.9.0 with apache-airflow-providers-elasticsearch==5.3.4 - same
Also checked dev-tools it says while pending:
Request URL:
https://url/api/v1/dags/Redis_Populate_Cities/dagRuns/manual__2024-05-03T15:35:16.283558+00:00/taskInstances/populate_redis_cities/logs/1?full_content=false
Referrer Policy:
strict-origin-when-cross-origin
It says when there's no log:
Request URL:
https://url/api/v1/dags/Redis_Populate_Cities/dagRuns/manual__2024-04-04T07:01:51.667463+00:00/taskInstances/populate_redis_cities/logs/1?full_content=false
Request Method:
GET
Status Code:
200 OK
Remote Address:
address:443
Referrer Policy:
strict-origin-when-cross-origin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "mrybas (via GitHub)" <gi...@apache.org>.
mrybas commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2431901533
Hi!
I have the same issue with
```
Apache Airflow
version | 2.9.3
Providers info
apache-airflow-providers-elasticsearch | 5.4.1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Mant1ss (via GitHub)" <gi...@apache.org>.
Mant1ss commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298253611
Checked more information from devtools:
- when task started i can get new lines from log (https://airflow/api/v1/dags/{}/dagRuns/{}/taskInstances/{}/logs/2?full_content=false)
- when response have more then 4k lines https://airflow/api/v1/dags/{}/dagRuns/{}/taskInstances/{}/logs/2?full_content=false can't get any new lines
- webserver show requests cycle (_count, _search):
```
[2024-08-20T07:31:35.390+0000] {_transport.py:349} INFO - POST https://*/_count [status:200 duration:0.033s]
[2024-08-20T07:31:35.425+0000] {_transport.py:349} INFO - POST https://*/_search [status:200 duration:0.034s]
```
- when end_of_log appear in elastic i can show full_log (previous request 4038 lines, last 67954)
- end_of_log exist not in all logs in elastic. I don't understand why. If end_of_log is not exist i can't view log on grid page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2297176957
cc @Owen-CH-Leung dis you encounter this problem with your latest work with elastic provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2085911669
Is it reproduced in the latest [`apache-airflow-providers-elasticsearch`](https://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/index.html)? And maybe also worthwhile to check on the latest Airflow ([2.9.0](https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html) at that moment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Mant1ss (via GitHub)" <gi...@apache.org>.
Mant1ss commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298799921
same issue https://github.com/apache/airflow/issues/28326
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "jedcunningham (via GitHub)" <gi...@apache.org>.
jedcunningham closed issue #39323: Logs from elastic not visible in Grid menu
URL: https://github.com/apache/airflow/issues/39323
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2084939643
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "ivanp221 (via GitHub)" <gi...@apache.org>.
ivanp221 commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2086146937
will test and report thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "rawwar (via GitHub)" <gi...@apache.org>.
rawwar commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2443265626
Noticed this issue for a BashOperator task. I'll edit this comment and more details later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298118503
No, the `Logs` tab are rendering fine. On main branch. I've enabled airflow remote logging with ES & I'm able to render logs from ES:
Here are some log outputs from webserver:
```
[2024-08-20T06:55:34.714+0000] {_transport.py:349} INFO - POST http://es01:9200/_all/_count [status:200 duration:0.008s]
[2024-08-20T06:55:34.719+0000] {_transport.py:349} INFO - POST http://es01:9200/_all/_search [status:200 duration:0.004s]
192.168.65.1 - - [20/Aug/2024:06:55:34 +0000] "GET /api/v1/dags/elasticsearch_sql_dag/dagRuns/manual__2024-08-20T06:52:41.314339+00:00/taskInstances/run_es_query/logs/1?full_content=false HTTP/1.1" 200 4887 "http://localhost:28080/dags/elasticsearch_sql_dag/grid?dag_run_id=manual__2024-08-20T06%3A52%3A41.314339%2B00%3A00&task_id=run_es_query&tab=logs" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Mant1ss (via GitHub)" <gi...@apache.org>.
Mant1ss commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298129644
@Owen-CH-Leung can you try long running task with 5k+ log lines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "YaAnuar (via GitHub)" <gi...@apache.org>.
YaAnuar commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2269165950
I solved this issue by adding new message into elasticsearch:
message: "end_of_log"
curl -X POST "http://elasticsearch-logs:9200/airflow-logs/_doc/" -H "Content-Type: application/json" -d '{
"log_id": "for_test_start_parse_2024_08_04T06_15_00_000000_1",
"message": "end_of_log",
"offset": 1
}'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "Owen-CH-Leung (via GitHub)" <gi...@apache.org>.
Owen-CH-Leung commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2298184697
In case it may help: I'm using simple curl command to upload the raw json logs to Elasticsearch (instead of filebeat) :
```
#/bin/bash
while IFS= read -r line
do
curl -X POST "http://localhost:9200/airflow-dag/_doc/" -H 'Content-Type: application/json' -d "$line"
done < [your DAG task log file]
```
Some important configs in `airflow.cfg` :
```
remote_logging = True
remote_log_conn_id = elasticsearch_default
[elasticsearch]
# Elasticsearch host
host = [your ES host]
json_format = True
offset_field = log
```
the raw task log file should look like the following:
```
{"asctime": "2024-08-20T06:52:42.617+0000", "filename": "local_task_job_runner.py", "lineno": 123, "levelname": "INFO", "message": "::group::Pre task execution logs", "log": 1724136762617922560, "dag_id": "elasticsearch_sql_dag", "task_id": "run_es_query", "execution_date": "2024_08_20T06_52_41_314339", "try_number": "1", "log_id": "elasticsearch_sql_dag-run_es_query-manual__2024-08-20T06:52:41.314339+00:00--1-1"}
```
If you are using tools like filebeat, be aware that not all keys in your raw log file will be attached as a field in a document inside ES. No matter what tool you use to write logs to ES, you need to make sure that a single document in ES looks like this:
```
{
"_index": "airflow-dag",
"_id": "o_ysbpEBEcTOdHp1B7dC",
"_score": 1,
"_source": {
"asctime": "2024-08-20T07:19:54.376+0000",
"filename": "logging_mixin.py",
"lineno": 190,
"levelname": "INFO",
"message": "row: ['docker-cluster', '.alerts-observability.slo.alerts-default', 'VIEW', 'ALIAS']",
"log": 1724138394376478200,
"dag_id": "elasticsearch_sql_dag",
"task_id": "run_es_query",
"execution_date": "2024_08_20T07_19_51_268943",
"try_number": "1",
"log_id": "elasticsearch_sql_dag-run_es_query-manual__2024-08-20T07:19:51.268943+00:00--1-1"
}
}
```
Make sure there're fields `log` and `log_id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "sjnov11 (via GitHub)" <gi...@apache.org>.
sjnov11 commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2158863069
Same happens to me.
i am using images 2.6.3-python3.11
Grid menu logs call this API:
- {host:port}/api/v1/dags/tutorial/dagRuns/scheduled__2024-06-08T00:00:00+00:00/taskInstances/print_date/logs/1
pod logs:
```
[2024-06-11T01:31:13.259+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:13.774+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:14.220+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:14.289+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:14.736+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:14.805+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
[2024-06-11T01:31:15.252+0900] {base.py:270} INFO - POST {host:port}/_all/_count [status:200 request:0.003s]
....
[2024-06-11 01:31:22 +0900] [109] [CRITICAL] WORKER TIMEOUT (pid:161)
[2024-06-11 01:31:22 +0900] [161] [INFO] Worker exiting (pid: 161)
[2024-06-11 01:31:23 +0900] [109] [WARNING] Worker with pid 161 was terminated due to signal 9
[2024-06-11 01:31:23 +0900] [168] [INFO] Booting worker with pid: 168
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Logs from elastic not visible in Grid menu [airflow]
Posted by "RNHTTR (via GitHub)" <gi...@apache.org>.
RNHTTR commented on issue #39323:
URL: https://github.com/apache/airflow/issues/39323#issuecomment-2087551208
It's also worth checking your browser's dev tools to see what happens when loading the page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org