You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kmaksimovv (via GitHub)" <gi...@apache.org> on 2025/06/05 15:28:40 UTC
[I] airflow no writing logs to Elasticsearch [airflow]
kmaksimovv opened a new issue, #51456:
URL: https://github.com/apache/airflow/issues/51456
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
Use apache-airflow-providers-elasticsearch 5.4.1
Airflow 2.9.2
Writing logs to Elasticsearch not worked, no data in Elasticsearch
Config:
elasticsearch:
host: "https://user:pass@elasticsearch_host:9200"
target_index: 'airflow-logs'
log_id_template: '{dag_id}_{task_id}_{execution_date}_{try_number}'
index_patterns: 'airflow-*'
end_of_log_mark: "end_of_log"
write_stdout: "False"
write_to_es: 'True'
json_format: 'True'
elasticsearch_configs:
use_ssl: 'True'
verify_certs: 'True'
ca_certs: /es_certs/ca.pem
### What you think should happen instead?
_No response_
### How to reproduce
Use apache-airflow-providers-elasticsearch 5.4.1
Airflow 2.9.2
Writing logs to Elasticsearch not worked, no data in Elasticsearch
Config:
elasticsearch:
host: "https://user:pass@elasticsearch_host:9200"
target_index: 'airflow-logs'
log_id_template: '{dag_id}_{task_id}_{execution_date}_{try_number}'
index_patterns: 'airflow-*'
end_of_log_mark: "end_of_log"
write_stdout: "False"
write_to_es: 'True'
json_format: 'True'
elasticsearch_configs:
use_ssl: 'True'
verify_certs: 'True'
ca_certs: /es_certs/ca.pem
### Operating System
debian
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "eff917 (via GitHub)" <gi...@apache.org>.
eff917 commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-3150583405
I have run into the same problem.
Tested:
airlfow helm chart, with kubernetesExecutor.
Airflow versions tried:
3.03, 2.11.0, 2.10.5, 2.9.3
Elasticsearch versions tried: 9.0.0, 8.19.0
Webserver correctly picks up ES connection, and queries return http 200 with 0 results obviously.
Task pods don't even initiate a connection to ES, have confirmed this with tcpdump on the ES pod's network interface.
Envvars are picked up, because if I change the write_stdout to `True`, task pods write the json lines to stdout in addition to the standard text output.
relevant part of helm chart:
```
config:
...
elasticsearch:
write_stdout: 'False'
write_to_es: 'True'
# target_index: 'airflow_tasks'
json_format: 'True'
# log_id_template: "{dag_id}_{task_id}_{execution_date}_{try_number}"
elasticsearch_configs:
max_retries: 3
timeout: 30
retry_timeout: 'True'
verify_certs: 'False'
```
If I can do any more testing, or need any additional info, let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "AryanK1511 (via GitHub)" <gi...@apache.org>.
AryanK1511 commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2947905239
I can look into this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "JainSahit (via GitHub)" <gi...@apache.org>.
JainSahit commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-3508932535
@kmaksimovv https://github.com/apache/airflow/blob/main/chart/values.yaml#L3147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2948280506
I tried updating the version airflow, also apache-airflow-providers-elasticsearch to last version, but it still doesn't work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "AryanK1511 (via GitHub)" <gi...@apache.org>.
AryanK1511 commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2954071481
Update: I am able to recreate issue, will start looking for a fix now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #51456: airflow no writing logs to Elasticsearch
URL: https://github.com/apache/airflow/issues/51456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-3024345464
did you manage to fix it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-3495940300
@JainSahit What do you mean logs persistence enabled ? How do I turn it on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "AryanK1511 (via GitHub)" <gi...@apache.org>.
AryanK1511 commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2949331260
> [@AryanK1511](https://github.com/AryanK1511) When can you watch?
Im sorry I don't think I understand your question 😅, could you elaborate a bit more please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 closed issue #51456: airflow no writing logs to Elasticsearch
URL: https://github.com/apache/airflow/issues/51456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2954834346
This is excellent!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2948276539
@AryanK1511 When can you watch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "JainSahit (via GitHub)" <gi...@apache.org>.
JainSahit commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-3495517773
@kmaksimovv Do you have logs persistence enabled?
I have gone through immense troubles in getting task logs persisted in ES (even when log persistence disabled). If this is what you are looking for, I can help you out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "kmaksimovv (via GitHub)" <gi...@apache.org>.
kmaksimovv commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2948156188
It would be nice if it worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "AryanK1511 (via GitHub)" <gi...@apache.org>.
AryanK1511 commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2949332743
> I tried updating the version airflow, also apache-airflow-providers-elasticsearch to last version(6.3.0), but it still doesn't work the same problem here [#50349 (comment)](https://github.com/apache/airflow/issues/50349#issuecomment-2870293144)
Yeah lemme try to recreate this problem first and then I will post my findings here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] airflow no writing logs to Elasticsearch [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #51456:
URL: https://github.com/apache/airflow/issues/51456#issuecomment-2944992457
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org