You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/05 12:20:42 UTC
[GitHub] [airflow] amiriry opened a new issue #17443: No link for kibana when using frontend configuration
amiriry opened a new issue #17443:
URL: https://github.com/apache/airflow/issues/17443
**Apache Airflow version**: 2.1.2
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): using docker-compose not kubernetes.
**Environment**:
- **Cloud provider or hardware configuration**: ubuntu 20.04.2
- **OS** (e.g. from /etc/os-release): Linux
- **Kernel** (e.g. `uname -a`):5.8.0-63-generic
- **Install tools**:
- **Others**:
**What happened**:
I used docker-compose to start an environment of Airflow with Elasticsearch.
URL: https://github.com/amiriry/DockerCompose/blob/master/AirflowESK/docker-compose.yaml
As You can see in the file I set the frontend configuration:
AIRFLOW__ELASTICSEARCH__FRONTEND: "http://10.2.0.12:5601/{log_id}"
As described in the documentation
**What you expected to happen**:
After running the basic dag in the repo https://github.com/amiriry/DockerCompose/blob/master/AirflowESK/dags/dag_with_id.py , when opening the status box of the run, there should be a link to the kibana that docker-compose brings up.
There isn't.
I tried a lot of other urls which didn't work too.
**How to reproduce it**:
Run the dag and get into status box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-893414516
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry edited a comment on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry edited a comment on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-894785991
I'm using version 2.1.2 of airflow and added the remote logging setting as you can see in the file in my repo.
The task handler that I am using is ElasticsearchTaskHandler handler:
`airflow info | grep 'task_logging_handler'`
output:
`task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler
Which is the one your mentioned.
And still doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-894564611
Also make sure you are on version 2.0.1 or later of `apache-airflow-providers-elasticsearch` as it was broken in earlier versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-906750182
@amiriry, this change did go out in 2.1.3. Have you had a chance to confirm your link is now present and working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-894785991
I'm using version 2.1.2 of airflow and added the remote logging setting as you can see in the file in my repo. but still it doesn't work.
The task handler that I am using is es_task handler:
`airflow info | grep 'task_logging_handler'
task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.Ela`
And still doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry edited a comment on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry edited a comment on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-907756755
When I try to run again with docker-compose, airflow doesn't go up and I get this error, couple of times:
```
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 18, in <module>
from . import _mysql
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 445, in initialize
configure_adapters()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 325, in configure_adapters
import MySQLdb.converters
File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 24, in <module>
version_info, _mysql.version_info, _mysql.__file__
NameError: name '_mysql' is not defined
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-895254586
Can run `airflow providers list` and see what version of `apache-airflow-providers-elasticsearch` you have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry edited a comment on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry edited a comment on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-894785991
I'm using version 2.1.2 of airflow and added the remote logging setting as you can see in the file in my repo.
The task handler that I am using is ElasticsearchTaskHandler handler:
`airflow info | grep 'task_logging_handler'`
output:
`task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler`
Which is the one your mentioned.
And still doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-893891221
To enable external links for logs, you need to use a task handler that supports external links. In this case, you should use https://github.com/apache/airflow/blob/866a601b76e219b3c043e1dbbc8fb22300866351/airflow/providers/elasticsearch/log/es_task_handler.py#L44
It looks like you haven't remote logging turned on, so this task handler is not used.
```
[logging]
remote_logging = True
```
To check the current task handler, you can use ``airflow info`` command:
```
$ airflow info | grep 'task_logging_handler'
task_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandler
```
For more details, see:
http://airflow.apache.org/docs/apache-airflow-providers-elasticsearch/stable/logging.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-907756755
When I try to run again I get this error, couple of times:
```
ERROR! Maximum number of retries (20) reached.
Last check result:
$ airflow db check
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 18, in <module>
from . import _mysql
ImportError: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: cannot allocate memory in static TLS block
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 5, in <module>
from airflow.__main__ import main
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
settings.initialize()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 445, in initialize
configure_adapters()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/settings.py", line 325, in configure_adapters
import MySQLdb.converters
File "/home/airflow/.local/lib/python3.8/site-packages/MySQLdb/__init__.py", line 24, in <module>
version_info, _mysql.version_info, _mysql.__file__
NameError: name '_mysql' is not defined
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry edited a comment on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry edited a comment on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-894785991
I'm using version 2.1.2 of airflow and added the remote logging setting as you can see in the file in my repo. but still it doesn't work.
The task handler that I am using is es_task handler:
`airflow info | grep 'task_logging_handler'`
output:
`task_logging_handler | airflow.providers.elasticsearch.log.es_task_handler.Ela`
And still doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-896237478
Ah, okay, I see what the issue is, thanks @amiriry. The core specific changes from #16357 never made it into a core release. I'm going to tag it for 2.1.3, hopefully it gets included there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amiriry commented on issue #17443: No link for kibana when using frontend configuration
Posted by GitBox <gi...@apache.org>.
amiriry commented on issue #17443:
URL: https://github.com/apache/airflow/issues/17443#issuecomment-895825612
I did the command:
`docker exec $(docker ps | awk '/worker/ {print $NF}') airflow providers list | grep elasticsearc`
The output:
`apache-airflow-providers-elasticsearch | Elasticsearch https://www.elastic.co/elasticsearch | 2.0.2`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org