You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/12 21:06:07 UTC
[GitHub] [airflow] marcusianlevine opened a new issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
marcusianlevine opened a new issue #11479:
URL: https://github.com/apache/airflow/issues/11479
**Apache Airflow version**: 1.10.12
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): 1.16.9
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**: Docker image running in k8s Pods
- **Others**: Rancher-provisioned k8s clusters
**What happened**:
After configuring the latest version of the Elasticsearch backport provider as my log handler via `config/airflow_local_settings.py` resulted in an error on the webserver when trying to read logs from Elasticsearch
```
[2020-10-12 21:02:00,487] {app.py:1892} ERROR - Exception on /get_logs_with_metadata [GET]
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py", line 121, in wrapper
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/flask_appbuilder/security/decorators.py", line 109, in wraps
return f(self, *args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/decorators.py", line 56, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/views.py", line 733, in get_logs_with_metadata
logs, metadata = _get_logs_with_metadata(try_number, metadata)
File "/usr/local/lib/python3.7/site-packages/airflow/www_rbac/views.py", line 724, in _get_logs_with_metadata
logs, metadatas = handler.read(ti, try_number, metadata=metadata)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/log/file_task_handler.py", line 194, in read
logs[i] += log
TypeError: can only concatenate str (not "list") to str
```
Here is the relevant section of my customized `airflow_local_settings.py` file with the updated Elasticsearch handler from the backport provider:
```
...
elif ELASTICSEARCH_HOST:
ELASTICSEARCH_LOG_ID_TEMPLATE: str = conf.get('elasticsearch', 'LOG_ID_TEMPLATE')
ELASTICSEARCH_END_OF_LOG_MARK: str = conf.get('elasticsearch', 'END_OF_LOG_MARK')
ELASTICSEARCH_FRONTEND: str = conf.get('elasticsearch', 'frontend')
ELASTICSEARCH_WRITE_STDOUT: bool = conf.getboolean('elasticsearch', 'WRITE_STDOUT')
ELASTICSEARCH_JSON_FORMAT: bool = conf.getboolean('elasticsearch', 'JSON_FORMAT')
ELASTICSEARCH_JSON_FIELDS: str = conf.get('elasticsearch', 'JSON_FIELDS')
ELASTIC_REMOTE_HANDLERS: Dict[str, Dict[str, Union[str, bool]]] = {
'task': {
'class': 'airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler',
'formatter': 'airflow',
'base_log_folder': str(os.path.expanduser(BASE_LOG_FOLDER)),
'log_id_template': ELASTICSEARCH_LOG_ID_TEMPLATE,
'filename_template': FILENAME_TEMPLATE,
'end_of_log_mark': ELASTICSEARCH_END_OF_LOG_MARK,
'host': ELASTICSEARCH_HOST,
'frontend': ELASTICSEARCH_FRONTEND,
'write_stdout': ELASTICSEARCH_WRITE_STDOUT,
'json_format': ELASTICSEARCH_JSON_FORMAT,
'json_fields': ELASTICSEARCH_JSON_FIELDS
},
}
LOGGING_CONFIG['handlers'].update(ELASTIC_REMOTE_HANDLERS)
...
```
**What you expected to happen**:
Airflow's web UI properly displays the logs from Elasticsearch
**How to reproduce it**:
Configure custom logging via `config/airflow_local_settings.py` to `airflow.providers.elasticsearch.log.es_task_handler.ElasticsearchTaskHandler` and set the `logging_config_class` in `airflow.cfg`
When a task has been run, try to view its logs in the web UI and check the webserver logs to see the error above
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-707399551
This looks like a real problem. Should I assign you to this issue? Do you want to deal with it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-707875552
Sure, I think that will work, I'll open a PR later today
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #11479:
URL: https://github.com/apache/airflow/issues/11479
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-730676103
Hey @marcusianlevine -> can you plese take a look at my comment there? I looked at the providers that are released and they seem to contain the right refactor, so I guess this is a problem with your local installation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-707402279
@marcusianlevine What do you think to vendorize the `file_task_handler` module to `elasticsearch` package? See: https://github.com/apache/airflow/blob/master/provider_packages/refactor_provider_packages.py#L427-L436
This will allow us to fix the problem without having to make any changes to Airflow core.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-707402279
@marcusianlevine What do you think to vendor the `file_task_handler` module to `elasticsearch` package? See: https://github.com/apache/airflow/blob/master/provider_packages/refactor_provider_packages.py#L427-L436
This will allow us to fix the problem without having to make any changes to Airflow core.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-730467228
Unfortunately it seems that my first PR didn't work - the refactor didn't take for some reason
https://github.com/apache/airflow/pull/11509#issuecomment-730441486
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marcusianlevine commented on issue #11479: Elasticsearch Backport Provider Incompatible with Airflow 1.10.12
Posted by GitBox <gi...@apache.org>.
marcusianlevine commented on issue #11479:
URL: https://github.com/apache/airflow/issues/11479#issuecomment-707400199
I can take a stab - I think we just need to backport some of the type handling from `airflow/utils/log/file_task_handler.py` on `master` to the 1.10 branch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org